Lead Data Engineer

2 weeks ago

Pune, Maharashtra, India ITC Infotech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Lead Data Engineer

Location: Banglore/Pune (Hybrid)

Mode: Hybrid

Shift Timing: 2 PM to 11 PM

Experience: 7-12 years.
3+ years in cloud data platforms and Databricks.

Purpose:

We are seeking a hands-on
Technical Lead
to drive the ingestion of high-volume mainframe RPC data into
Databricks
, enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.

Key Responsibilities

  • Mainframe Data Ingestion:
  • Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.
  • ML-Ready Data Engineering:
  • Transform and structure data for time-series modelling and anomaly detection across thousands of models.
  • ML Workflow Integration:
  • Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types; enable automated retraining and scoring.
  • Performance Optimization:
  • Tune Spark jobs, Delta Lake storage, and cluster configurations for billions of records and real-time aggregation.
  • FinOps & Cost Control:
  • Monitor and optimize Databricks resource usage; implement auto-scaling and cost-aware job scheduling.
  • Monitoring & Alerting:
  • Integrate Databricks-native alerting for pipeline health, data anomalies, and job failures.

Required Skills

  • Strong hands-on experience with
    Databricks
    ,
    Apache Spark
    ,
    Delta Lake
    , and
    MLflow
  • Proven expertise in
    mainframe data integration
    (e.g., SMF, RPC logs, VSAM, DB2)
  • Strong Python and PySpark programming skills

Preferred Skills

  • Databricks experience with
    CI/CD tools
    GitHub Actions
  • Knowledge of
    FinOps principles

for cloud cost optimization

  • Lead Data Engineer

    2 days ago

    Pune, Maharashtra, India EXL Talent Acquisition Team Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Lead Data Engineer Job DescriptionRole Overview:We are looking for a skilled Lead Data Engineer to design, build, and manage scalable data pipelines and architectures. You will lead a team of data engineers, collaborate with cross-functional stakeholders, and drive best practices in data engineering to support data-driven decision-making.Key...

  • Lead Data Engineer

    1 day ago

    Pune, Maharashtra, India Jet2 Travel Technologies Pvt Ltd. Full time ₹ 25,00,000 - ₹ 1,00,00,000 per year

    Job Title: Lead Data EngineerJob Type: PermanentRelevant Experience: 8+ years experience  in a data-related role with recent experience in the field of cloud data engineering. Experience of cloud data analytics and data warehousing would be an advantage.Reports to: Engineering Manager – Data & AnalyticsJob Description:We, at Jet2 (UK's third largest...

  • Data Engineer Lead

    6 days ago

    Pune, Maharashtra, India Mappy Global Resources Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Greetings for the DayScouting for a Director, Data Engineering Lead to be associated with a Global Digital Media Agency.Location: PuneShift: EMEARole & responsibilitiesDirector, Data Engineering Lead, you will drive the architecture, design, and implementation of enterprise-scale data platforms and pipelines. You will lead a high-performing team of data...

  • Lead Software Engineer

    4 days ago

    Pune, Maharashtra, India Data Direct Networks Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    OverviewThis is an incredible opportunity to be part of a company that has been at the forefront of AI and high-performance data storage innovation for over two decades. DataDirect Networks (DDN) is a global market leader renowned for powering many of the world's most demanding AI data centers, in industries ranging from life sciences and healthcare to...

  • Lead Software Engineer

    2 weeks ago

    Pune, Maharashtra, India Data Direct Networks Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    OverviewThis is an incredible opportunity to be part of a company that has been at the forefront of AI and high-performance data storage innovation for over two decades. DataDirect Networks (DDN) is a global market leader renowned for powering many of the world's most demanding AI data centers, in industries ranging from life sciences and healthcare to...

  • Lead Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Persistent Systems Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    About Position:We're looking for a Lead Technical Engineer who can bridge data engineering depth with FinOps insight. You'll design and implement scalable data solutions across Databricks and Snowflake, drive performance optimization, and ensure that our cloud operations stay cost-efficient and well-governed.Role: Lead Data EngineerLocation: All Persistent...

  • Lead Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Persistent Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About Position:Were looking for a Lead Technical Engineer who can bridge data engineering depth with FinOps insight. You'll design and implement scalable data solutions across Databricks and Snowflake, drive performance optimization, and ensure that our cloud operations stay cost-efficient and well-governed.Role: Lead Data EngineerLocation: All Persistent...

  • Lead Data Engineer

    1 week ago

    Pune, Maharashtra, India Shashwath Solution Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We are seeking a skilled Lead Data Engineer with extensive experience in Snowflake, ADF, SQL, and other relevant data technologies to join our team. As a key member of our data engineering team, you will play an instrumental role in designing, developing, and managing data pipelines, working closely with cross-functional teams to drive the success of our...

  • Lead Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Searce Inc Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Who we are?Searce means 'a fine sieve' & indicates 'to refine, to analyze, to improve'. It signifies our way of working: To improve to the finest degree of excellence, 'solving for better' every time. Searcians are passionate improvers & solvers who love to question the status quo.The primary purpose of all of us, at Searce, is driving intelligent, impactful...

  • Data Engineering Lead

    2 weeks ago

    Pune, Maharashtra, India SiteMinder Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    At SiteMinder we believe the individual contributions of our employees are what drive our success. That's why we hire and encourage diverse teams that include and respect a variety of voices, identities, backgrounds, experiences and perspectives. Our diverse and inclusive culture enables our employees to bring their unique selves to work and be proud of...