Lead Azure Databricks Data Engineer

19 hours ago

Bhopal, Madhya Pradesh, India SAA Consultancy Full time ₹ 1,04,000 - ₹ 13,08,780 per year

Location: Bhopal, MP, India

Experience: 7+ years


Key Responsibilities

  • Design and build ETL pipelines in Azure Databricks (PySpark, Delta Lake) to load, clean, and deliver data across Bronze, Silver, and Gold layers.
  • Implement Data Lakehouse Architecture on Azure Data Lake Gen2 with partitioning, schema management, and performance optimization.
  • Develop data models (dimensional/star schema) for reporting in Synapse and Power BI.
  • Integrate Databricks with Azure services – ADF, Key Vault, Event Hub, Synapse Analytics, Purview, and Logic Apps.
  • Build and manage CI/CD pipelines in Azure DevOps (YAML, Git repos, pipelines).
  • Optimize performance through cluster tuning, caching, Z-ordering, Delta optimization, and job parallelization.
  • Ensure data security and compliance (row-level security, PII masking, GDPR/HIPAA, audit logging).
  • Collaborate with data architects and analysts to translate business needs into technical solutions.

Required Skills

  • Strong experience in Azure Databricks (Python, PySpark, SQL).
  • Proficiency with Delta Lake (ACID transactions, schema evolution, incremental loads).
  • Hands-on with Azure ecosystem – Data Factory, ADLS Gen2, Key Vault, Event Hub, Synapse.
  • Knowledge of data governance & lineage tools (Purview, Unity Catalog is a plus).
  • Strong understanding of data warehouse design and star schema.
  • Azure DevOps (YAML, Git repos, pipelines) experience.
  • Good debugging skills for performance tuning & schema drift issues.

Good to Have

  • Experience with healthcare or financial data.
  • Familiarity with FHIR, OMOP, OpenEHR (for healthcare projects).
  • Exposure to AI/ML integration using Databricks ML runtime.
  • Experience with Unity Catalog for governance across workspaces.

What You Will Deliver

  • Automated snapshot + incremental pipelines in Databricks.
  • Delta Lake architecture with partitioning, Z-ordering, and schema drift handling.
  • Metadata-driven ingestion framework (YAML configs).
  • Power BI datasets connected to Gold layer.
  • CI/CD pipelines for deployment across.

Are you ready to take the lead in building scalable data solutions with Azure Databricks? Apply now

  • azure data engineer

    4 days ago

    Bhopal, Madhya Pradesh, India Eagle in Cloud Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    WE ARE HIRINGJob Role- Azure Data EngineerExperience - 7+ YearsLocation - IndoreNotice Period - Immediate JoinersPrimary Skills: Advanced SQL Server, ETL development using SSIS, Azure Data Factory, and Databricks, Power BI, Data warehousing, Data quality and validation, General Microsoft technology stackRole DescriptionThis is a full-time on-site role...

  • Presales Solution Architect

    4 days ago

    Bhopal, Madhya Pradesh, India Intertec Systems Full time

    Skills : Solution Architecture, Data Platform Design, Cloud Platforms, Pre-sales Consulting, Microsoft Fabric, Azure Synapse Analytics, Azure Data Lake, Azure SQL,Job Description : We are seeking an experienced Presales Solution Architect with deep Microsoft Data Platform & AI expertise, combined with knowledge and exposure to other modern data platforms...

  • Data Engineer

    3 weeks ago

    Bhopal, Madhya Pradesh, India Katyayani Organics Full time

    Position Data Engineer Location Bhopal Madhya Pradesh Experience Required 2-5 Years in Data Engineering Industry Chemical Manufacturing Website www katyayaniorganics com About Katyayani Organics At Katyayani Organics we are a rapidly growing leader in agro-inputs Established in 2016 we specialize in organic raw materials pesticides ...

  • Data Engineering Specialist

    4 days ago

    Bhopal, Madhya Pradesh, India beBeeExpert Full time ₹ 12,00,000 - ₹ 20,00,000

    Analytics and Reporting ExpertWe are seeking a highly skilled Analytics and Reporting Expert to develop and maintain cutting-edge analytics and reporting applications. The ideal candidate should have extensive experience working with Python (Flask), Go, and modern reporting platforms. Strong familiarity with Azure and relational databases is essential. Key...

  • Tech Lead

    5 days ago

    Bhopal, Madhya Pradesh, India Bhautiki Plus Edutech Pvt. Ltd. Full time

    About Us Bhautiki Plus is an EdTech company based in Gwalior, focused on IIT-JEE, NEET, and Foundation courses. We are building scalable technology platforms to enhance student learning, mentor efficiency, and parent engagement. We seek a Tech Lead to drive the design and development of our web applications with modern, reliable, and scalable...

  • Data Extraction Engineer

    3 days ago

    Bhopal, Madhya Pradesh, India Relu Consultancy Full time ₹ 50,000 - ₹ 28,00,000 per year

    Relu Consultancy is seeking a Data Extraction Engineer with expertise in Python (Selenium). In this role, you will design, implement, and maintain robust data scraping solutions that drive our projects forward. This is your chance to contribute to cutting-edge initiatives while enjoying a work schedule tailored to your needs.Job Title:Data Extraction...

  • IOT engineer

    4 days ago

    Bhopal, Madhya Pradesh, India Weekday AI Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    This role is for one of the Weekday's clientsSalary range: Rs Rs ie INR 5-8 LPA)Min Experience: 3 yearsJobType: full-timeWe are seeking a skilled IoT Engineer with strong expertise in developing and deploying IoT-based solutions. The ideal candidate will have hands-on experience with sensor integration, device connectivity, real-time monitoring, and...

  • IOT engineer

    4 days ago

    Bhopal, Madhya Pradesh, India Weekday AI Full time ₹ 5,00,000 - ₹ 8,00,000

    This role is for one of the Weekday's clientsSalary range: Rs Rs ie INR 5-8 LPA)Min Experience: 3 yearsJobType: full-timeWe are seeking a skilled IoT Engineer with strong expertise in developing and deploying IoT-based solutions. The ideal candidate will have hands-on experience with sensor integration, device connectivity, real-time monitoring, and...

  • IOT engineer

    4 days ago

    Bhopal, Madhya Pradesh, India Weekday AI Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    This role is for one of the Weekday's clientsSalary range: Rs Rs ie INR 5-8 LPA)Min Experience: 3 yearsJobType: full-time We are seeking a skilled IoT Engineer with strong expertise in developing and deploying IoT-based solutions. The ideal candidate will have hands-on experience with sensor integration, device connectivity, real-time monitoring, and...

  • Leading Biofuel Plant Engineer

    6 days ago

    Bhopal, Madhya Pradesh, India beBeeEngineering Full time ₹ 1,04,000 - ₹ 1,30,878

    Lead Biofuel Plant EngineerWe are seeking a highly experienced Lead Biofuel Plant Engineer to oversee the setup, commissioning and operations of our biofuel facility.Job DescriptionThe successful candidate will lead daily plant setup and operations to achieve optimal performance. This includes optimizing digestion, gas cleanup/upgrading, compression,...