Data Engineer lead

2 weeks ago

Bengaluru, Karnataka, India Innova ESI Full time ₹ 1,04,000 - ₹ 1,30,878 per year

Role: Data Engineer lead

Experience: 10+ Years

Location: Bangalore

Notice: Immediate Joiners

Job Description:

Job Description – Data Engineer (Azure, ADF, Databricks, PySpark, SCD, Unity Catalog, SQL)

Role Overview:

Looking for highly skilled experienced Data Engineer with expertise in Azure Data Factory (ADF), Azure Databricks, Delta Tables, Unity Catalog, Slowly Changing Dimension Type 2 (SCD2), and PySpark. The ideal candidate will be responsible for designing, developing, and optimizing data pipelines and ETL workflows while ensuring data integrity, scalability, and security within the Azure ecosystem.

Key Responsibilities:

  1. Develop and optimize data pipelines using Azure Data Factory (ADF) and Azure Databricks for large-scale data processing.
  2. Implement Slowly Changing Dimension in Delta Tables to manage historical data changes effectively.
  3. Leverage Unity Catalog for secure and organized data governance, cataloging, and access control across Databricks.
  4. Write efficient PySpark code to process and transform large datasets, ensuring high performance and scalability.
  5. Design and implement ETL/ELT solutions to integrate data from multiple sources into Delta Lake.
  6. Monitor, debug, and optimize existing data pipelines to ensure smooth operations and minimal downtime.
  7. Ensure data quality, consistency, and lineage tracking through best practices and automation.
  8. Collaborate with data architects, analysts, and business teams to define requirements and implement data-driven solutions.

Required Skills & Qualifications:

  1. 10+ years of experience in Data Engineering with a focus on Azure technologies.
  2. Expertise in Azure Data Factory (ADF) & Azure Databricks for ETL/ELT workflows.
  3. Strong knowledge of Delta Tables & Unity Catalog for efficient data storage and management.
  4. Experience with Slowly Changing Dimensions (SCD2) implementation in Delta Lake.
  5. Proficiency in PySpark for large-scale data processing & transformation.
  6. Hands-on experience with SQL & performance tuning for data pipelines.
  7. Understanding of data governance, security, and compliance best practices in Azure.
  8. Knowledge of CI/CD, DevOps practices for data pipeline automation.

Preferred Qualifications:

  1. Experience with
    Azure Synapse Analytics, Data Lakes, and Power BI integration
    .
  2. Knowledge of
    Kafka or Event Hub
    for real-time data ingestion.
  3. Certifications in
    Microsoft Azure (DP-203, DP-900) or Databricks
    are a plus.

  • Data Engineer

    1 day ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer...

  • Lead Data Scientist

    5 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Lead Data Scientist - Computer Vision & Generative AI to join our team in Bangalore, Karnātaka (IN-KA), India (IN). We are seeking...

  • Associate Lead Data Scientist

    4 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Sigmoid Analytics is a leading Data solutions company backed by Sequoia Capital. We offer best in- end-to-end data value chain spanning across Data Science, Data Engineering and Data Ops. With data and technology at the core of our solutions, we are solving some of the toughest problems out there. Our culture is modelled around expertise and mutual respect...

  • Sr. Data Engineer Data Science

    2 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Req ID: 342869NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Sr. Data Engineer Data Science to join our team in Bangalore, Karnātaka (IN-KA), India (IN). ...

  • Lead Data Scientist

    2 weeks ago

    Bengaluru, Karnataka, India Aptus Data Labs Full time US$ 1,25,000 - US$ 1,75,000 per year

    Job Description: Senior/lead Data ScientistLocation: Bangalore/WFHEmployment Type: Full-timeAbout the RoleWe are seeking a highly skilled Senior /lead Data Scientist with strong expertise in Artificial Intelligence (AI), Machine Learning (ML), Deep Learning, and Natural Language Processing (NLP). The ideal candidate will also have hands-on exposure to...

  • Lead Data Scientist

    2 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    We are currently seeking a Lead Data Scientist - Computer Vision & Generative AI to join our team in Bangalore, Karntaka (IN-KA), India (IN).We are seeking a highly skilled and experienced Lead Data Scientist with 7+ years of expertise in Machine Learning, Deep Learning (especially Computer Vision), and Generative AI. The ideal candidate will lead the design...

  • Data Engineer

    1 day ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key Responsibilities:Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from...

  • Data Engineer

    7 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    As a Data Engineer, your main responsibilities will involve: The project is a migration of a SAS code to DBT code, so it's important that the candidate has experience in this type of projects. Analyze data models and derive logical conclusions. Processes modeling. Hands on development and monitoring of the Azure cloud Platform and various associated...

  • Data Engineer

    2 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...

  • Data Engineer

    7 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 8,00,000 - ₹ 16,00,000 per year

    Build scalable data pipelines from structured and unstructured sources Design ontologies that model business entities, relationships, and process flows Collaborate with domain SMEs and ML engineers to structure agent inputs and outputs Implement transformation, validation, and enrichment logic across diverse datasets Maintain and optimize cloud data...