Data Engineer lead

2 weeks ago

Bengaluru, India Innova ESI Full time

Role: Data Engineer lead

Experience: 10+ Years

Location: Bangalore

Notice: Immediate Joiners

Job Description:

Job Description – Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL)

Role Overview:

Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog, Slowly Changing Dimension Type 2 (SCD2), and PySpark. The ideal candidate will be responsible for designing, developing, and optimizing data pipelines and ETL workflows while ensuring data integrity, scalability, and security within the Azure ecosystem.

Key Responsibilities:

  1. Develop and optimize data pipelines using Azure Data Factory (ADF) and Azure Databricks for large-scale data processing.
  2. Implement Slowly Changing Dimension in Delta Tables to manage historical data changes effectively.
  3. Leverage Unity Catalog for secure and organized data governance, cataloging, and access control across Databricks.
  4. Write efficient PySpark code to process and transform large datasets, ensuring high performance and scalability.
  5. Design and implement ETL/ELT solutions to integrate data from multiple sources into Delta Lake.
  6. Monitor, debug, and optimize existing data pipelines to ensure smooth operations and minimal downtime.
  7. Ensure data quality, consistency, and lineage tracking through best practices and automation.
  8. Collaborate with data architects, analysts, and business teams to define requirements and implement data-driven solutions.

Required Skills & Qualifications:

  1. 10+ years of experience in Data Engineering with a focus on Azure technologies.
  2. Expertise in Azure Data Factory (ADF) & Azure Databricks for ETL/ELT workflows.
  3. Strong knowledge of Delta Tables & Unity Catalog for efficient data storage and management.
  4. Experience with Slowly Changing Dimensions (SCD2) implementation in Delta Lake.
  5. Proficiency in PySpark for large-scale data processing & transformation.
  6. Hands-on experience with SQL & performance tuning for data pipelines.
  7. Understanding of data governance, security, and compliance best practices in Azure.
  8. Knowledge of CI/CD, DevOps practices for data pipeline automation.

Preferred Qualifications:

  1. Experience with
    Azure Synapse Analytics, Data Lakes, and Power BI integration
    .
  2. Knowledge of
    Kafka or Event Hub
    for real-time data ingestion.
  3. Certifications in
    Microsoft Azure (DP-203, DP-900) or Databricks
    are a plus.

  • Technical Lead- Data Engineer

    7 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Kindly find the Job Description Below.Job Title:Technical Lead- Data EngineerLocation: BangaloreYears of Experience: 8+ years of experienceSigmoidworks with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software...

  • Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Data Analytics Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Get to Know the Team Grab has been expanding into multiple lines of businesses. The Grab Data Warehouse team builds business intelligence systems using data from all business lines to provide insights to key stakeholders across the company. Get to Know the Role We are looking for a Lead Data Engineer to lead the design and enhancement of our data...

  • Lead Data Scientist

    3 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Lead Data Scientist - Computer Vision Generative AI to join our team in Bangalore Karn taka IN-KA India IN We are seeking a...

  • Lead Data Scientist

    2 weeks ago

    Bengaluru, India NTT DATA Full time

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Lead Data Scientist - Computer Vision & Generative AI to join our team in Bangalore, Karnātaka (IN-KA), India (IN). We are...

  • Digital Engineering Lead Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Req ID 299336NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Digital Engineering Lead Engineer to join our team in Bangalore Karn xc4 x81taka IN-KA India IN How We Will Help...

  • Mid Data Science

    1 day ago

    Bengaluru, India NTT DATA Full time

    Job Description Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Mid Data Science & AIML, GenAI Lead/Engineer to join our team in Bangalore, Karnātaka (IN-KA), India...

  • Sr. Data Science

    1 day ago

    Bengaluru, India NTT DATA Full time

    Job Description Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Sr. Data Science & AIML, GenAI Lead/Engineer to join our team in Bangalore, Karnātaka (IN-KA), India...

  • Lead Data Scientist

    4 days ago

    Bengaluru, Karnataka, India Aptus Data Labs Full time US$ 1,25,000 - US$ 1,75,000 per year

    Job Description: Senior/lead Data ScientistLocation: Bangalore/WFHEmployment Type: Full-timeAbout the RoleWe are seeking a highly skilled Senior /lead Data Scientist with strong expertise in Artificial Intelligence (AI), Machine Learning (ML), Deep Learning, and Natural Language Processing (NLP). The ideal candidate will also have hands-on exposure to...

  • Lead Data Scientist

    1 week ago

    Bengaluru, Karnataka, India Enable Data Incorporated Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Enable Data Incorporated is looking for a talented and experienced Lead Data Scientist to join our innovative team. In this role, you will be responsible for driving data science initiatives, developing predictive models, and providing insights that guide key business decisions. The ideal candidate has a strong background in statistical analysis, machine...

  • Lead Data Scientist

    2 weeks ago

    Bengaluru, India Enable Data Incorporated Full time

    Enable Data Incorporated is looking for a talented and experienced Lead Data Scientist to join our innovative team. In this role, you will be responsible for driving data science initiatives, developing predictive models, and providing insights that guide key business decisions. The ideal candidate has a strong background in statistical analysis, machine...