Senior Data Platform Engineer

Position Summary As the Senior Data Platform Engineer Azure Databricks you will be responsible for building maintaining and evolving our Azure Databricks platform ensuring it is secure reliable scalable and cost-efficient This role is pivotal in empowering our Data- AIOps team by providing a robust automated and well-governed data environment The ideal candidate will drive a platform-as-a-product mindset continuously improving our architecture operations and user experience Your Responsibilities Platform Architecture Engineering Design and implement architectural improvements including workspace migrations and robust disaster recovery DR strategies Enhance and maintain the platform s security posture focusing on data governance data and compute isolation at logical and physical levels Drive the evolution of the platform by researching prototyping and integrating new tools and features to enhance performance and scalability Data Governance Security Lead the expansion and utilization of Databricks Unity Catalog for granular data governance including row column-level security data lineage tracking and metadata tagging Conduct regular security and compliance checks to ensure the platform adheres to internal policies and industry standards Automation Operations DevOps GitOps Build and maintain resilient CI CD pipelines using GitHub Actions to automate infrastructure Terraform and application deployments Implement and manage a comprehensive observability stack logging monitoring tracing alerting to ensure high availability and proactive issue resolution Apply FinOps principles to monitor analyze and optimize platform costs Manage day-to-day platform administration including onboarding projects managing clusters and configuring jobs User Enablement Self-Service Develop and integrate user-friendly interfaces e g via ServiceNow or Backstage to enable self-service for access requests and resource provisioning Create and maintain clear documentation including architecture diagrams runbooks and best-practice guides Coach and train other colleagues championing a platform-as-a-product mindset focused on delivering value to internal users Your profile Essential Qualifications Bachelor s or Master s degree in Computer Science Mathematics Engineering or a related discipline 5 years of professional experience in a platform engineering DevOps or SRE role with a focus on data-intensive platforms Deep expertise in Microsoft Azure including ADLSGen2 Private Endpoints Azure Key Vault Azure Log Analytics and EntraID Extensive hands-on experience with Azure Databricks administration operations security and performance tuning Strong proficiency with Terraform and a solid understanding of GitOps principles Advanced skills in shell scripting and experience with a scripting language like Python Demonstrable experience building CI CD pipelines with GitHub Actions Familiarity with Databricks Asset Bundles is a plus Proficient with Linux administration and troubleshooting Practical experience implementing data governance solutions for data lakes especially with Unity Catalog Solid understanding of Kubernetes k8s concepts and architecture Preferred Qualifications Understanding of MLOps principles e g feature stores model deployment strategies to support ML engineering teams Experience integrating with or building plugins for platforms like Backstage or ServiceNow Soft Skills and Cultural Fit Excellent communication and collaboration skills to work effectively with technical teams and business stakeholders Strong analytical thinking problem-solving abilities Demonstrated high level of initiative self-motivation and a proactive self-starter mindset with a strong drive to independently identify and solve challenges A focus on treating the platform as a product prioritizing the user experience and delivering tangible value to internal customers A passion for continuous learning innovation knowledge sharing and driving excellence in data engineering Ability to work effectively in a cross-functional fast-paced environment