Senior Data Platform Engineer

Position Summary :  As the (Senior) Data Platform Engineer (Azure Databricks) , you will be responsible for building, maintaining, and evolving our Azure Databricks platform, ensuring it is secure, reliable, scalable, and cost-efficient. This role is pivotal in empowering our Data- & AIOps team by providing a robust, automated, and well-governed data environment. The ideal candidate will drive a "platform-as-a-product" mindset, continuously improving our architecture, operations, and user experience. Your Responsibilities :  Platform Architecture & Engineering: Design and implement architectural improvements, including workspace migrations and robust disaster recovery (DR) strategies. Enhance and maintain the platform's security posture, focusing on data governance, data and compute isolation at logical and physical levels. Drive the evolution of the platform by researching, prototyping, and integrating new tools and features to enhance performance and scalability. Data Governance & Security: Lead the expansion and utilization of Databricks Unity Catalog for granular data governance, including row/column-level security, data lineage tracking, and metadata tagging. Conduct regular security and compliance checks to ensure the platform adheres to internal policies and industry standards. Automation & Operations (DevOps & GitOps): Build and maintain resilient CI/CD pipelines using GitHub Actions to automate infrastructure (Terraform) and application deployments. Implement and manage a comprehensive observability stack (logging, monitoring, tracing, alerting) to ensure high availability and proactive issue resolution. Apply FinOps principles to monitor, analyze, and optimize platform costs. Manage day-to-day platform administration, including onboarding projects, managing clusters, and configuring jobs. User Enablement & Self-Service: Develop and integrate user-friendly interfaces (e.g., via ServiceNow or Backstage) to enable self-service for access requests and resource provisioning. Create and maintain clear documentation, including architecture diagrams, runbooks, and best-practice guides. Coach and train other colleagues, championing a "platform-as-a-product" mindset focused on delivering value to internal users. Your profile Essential Qualifications: Bachelor's or Master's degree in Computer Science, Mathematics, Engineering, or a related discipline. 5+ years of professional experience in a platform engineering, DevOps, or SRE role with a focus on data-intensive platforms. Deep expertise in Microsoft Azure, including ADLSGen2, Private Endpoints, Azure Key Vault, Azure Log Analytics, and EntraID. Extensive hands-on experience with Azure Databricks administration, operations, security, and performance tuning. Strong proficiency with Terraform and a solid understanding of GitOps principles. Advanced skills in shell scripting and experience with a scripting language like Python. Demonstrable experience building CI/CD pipelines with GitHub Actions. Familiarity with Databricks Asset Bundles is a plus. Proficient with Linux administration and troubleshooting. Practical experience implementing data governance solutions for data lakes, especially with Unity Catalog. Solid understanding of Kubernetes (k8s) concepts and architecture. Preferred Qualifications: Understanding of MLOps principles (e.g., feature stores, model deployment strategies) to support ML engineering teams. Experience integrating with or building plugins for platforms like Backstage or ServiceNow. Soft Skills and Cultural Fit: Excellent communication and collaboration skills to work effectively with technical teams and business stakeholders. Strong analytical thinking, problem-solving abilities. Demonstrated high level of initiative, self-motivation, and a proactive, self-starter mindset, with a strong drive to independently identify and solve challenges. A focus on treating the platform as a product, prioritizing the user experience and delivering tangible value to internal customers. A passion for continuous learning, innovation, knowledge sharing, and driving excellence in data engineering. Ability to work effectively in a cross-functional, fast-paced environment.