Lead Databricks Engineer

We’re Hiring: Senior or Lead Databricks Developer Location : Baner, Pune (Onsite, 5 days a week) Work Mode : Full-time | Office-based ⏳ Availability : Immediate Joiners / Notice Period ≤ 30 Days ⚠️ Note : Only local Pune candidates may apply. * L2 interviews will be conducted in person — apply only if you are open to face-to-face interviews. About Newscape Consulting- Newscape is a fast-growing digital services company specializing in healthcare technology and data engineering. We help healthcare organizations modernize legacy systems, integrate cloud-native platforms, and drive digital transformation through data. If you’re passionate about solving complex data challenges and contributing to meaningful innovation in healthcare, this is the place for you. Position Summary - This role requires a seasoned Databricks professional with strong technical depth and leadership capability. You will design, architect, and optimize data pipelines using Databricks, Apache Spark, Delta Lake, and cloud-native services (Azure, AWS, or GCP). As a Senior or Lead engineer, you will guide best practices, mentor junior engineers, and collaborate closely with data architects, data scientists, and business stakeholders. Key Responsibilities Architect, design, and maintain large-scale data pipelines using Databricks, PySpark, Spark SQL, and optionally Scala. Lead the adoption and implementation of Databricks components including Workspace, Jobs, DLT (Delta Live Tables), Repos, and Unity Catalog. Build and optimize Delta Lake solutions aligned with Lakehouse and Medallion architecture standards. Partner with architects, engineering teams, and business stakeholders to translate complex business requirements into robust technical solutions. Establish coding standards, best practices, and reusable components for Databricks-based development. Drive CI/CD automation for Databricks deployments using Azure DevOps, GitHub Actions, or equivalent tools. Ensure end-to-end data governance, data quality, metadata management, and lineage tracking leveraging Unity Catalog or Azure Purview. Utilize orchestration tools such as Apache Airflow, Azure Data Factory, or Databricks Workflows for scheduling and monitoring. Conduct performance tuning for Spark jobs, cluster optimization, and cost-efficient compute strategies. Guide data modeling, SQL transformations, and data warehousing implementation using industry-standard best practices. Mentor and provide technical leadership to junior and mid-level data engineers. Participate in architecture reviews, solution design discussions, and roadmap planning. Required Skills and Qualifications Strong hands-on experience in data engineering with specific and extensive expertise in Databricks. Deep proficiency with Databricks Workspace, Jobs, DLT, Repos, Unity Catalog, and Lakehouse architecture. Advanced programming experience with PySpark and Spark SQL; Scala is an added advantage. Strong experience with at least one major cloud platform: Azure, AWS, or GCP. Proven expertise with Delta Lake, Lakehouse, and Medallion architecture patterns. Hands-on experience implementing CI/CD pipelines for Databricks using DevOps tooling. Familiarity with orchestration tools such as Airflow, ADF, or Databricks Workflows. Strong understanding of data governance, security frameworks, and enterprise data management. Experience integrating Databricks datasets with Power BI, including creating optimized data models, implementing Direct Lake or DirectQuery connections, and enabling performant, enterprise-grade reporting solutions. Experience leading technical teams or acting as a senior SME in critical data projects. Excellent analytical abilities, communication skills, and stakeholder management capability. Prior experience in healthcare data environments is a plus (not mandatory). Work Model This is a full-time, onsite role based in Baner, Pune. 5 days onsite per week is mandatory. Ready to make an impact? Apply now or DM us if you’re a Pune-based data enthusiast ready for your next big challenge How to Apply: Send your resume to swapnil.supe@newscapeconsulting.com Subject Line: Application – Azure Databricks Developer +91 8233829595