Lead Data Engineer

2 weeks ago

Pune, Maharashtra, India ITC Infotech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Lead Data Engineer

Location: Banglore/Pune (Hybrid)

Mode: Hybrid

Shift Timing: 2 PM to 11 PM

Experience: 7-12 years.
3+ years in cloud data platforms and Databricks.

Purpose:

We are seeking a hands-on
Technical Lead
to drive the ingestion of high-volume mainframe RPC data into
Databricks
, enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.

Key Responsibilities

  • Mainframe Data Ingestion:
  • Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.
  • ML-Ready Data Engineering:
  • Transform and structure data for time-series modelling and anomaly detection across thousands of models.
  • ML Workflow Integration:
  • Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types; enable automated retraining and scoring.
  • Performance Optimization:
  • Tune Spark jobs, Delta Lake storage, and cluster configurations for billions of records and real-time aggregation.
  • FinOps & Cost Control:
  • Monitor and optimize Databricks resource usage; implement auto-scaling and cost-aware job scheduling.
  • Monitoring & Alerting:
  • Integrate Databricks-native alerting for pipeline health, data anomalies, and job failures.

Required Skills

  • Strong hands-on experience with
    Databricks
    ,
    Apache Spark
    ,
    Delta Lake
    , and
    MLflow
  • Proven expertise in
    mainframe data integration
    (e.g., SMF, RPC logs, VSAM, DB2)
  • Strong Python and PySpark programming skills

Preferred Skills

  • Databricks experience with
    CI/CD tools
    GitHub Actions
  • Knowledge of
    FinOps principles

for cloud cost optimization

  • Data Engineer

    7 days ago

    Pune, Maharashtra, India Jash Data Sciences Full time

    Do you love solving real-world data problems with the latest and best techniques? And having fun while solving them in a team Then come join our high-energy team of passionate data people. Jash Data Sciences is the right place for you.We are a cutting-edge Data Sciences and Data Engineering startup based in Pune, India.We believe in continuous learning and...

  • Lead Data Engineer

    2 days ago

    Pune, Maharashtra, India Jet2 Travel Technologies Pvt Ltd. Full time

    Job Title: Lead Data EngineerJob Type: PermanentRelevant Experience: 8+ years experience  in a data-related role with recent experience in the field of cloud data engineering. Experience of cloud data analytics and data warehousing would be an advantage.Reports to: Engineering Manager – Data & AnalyticsJob Description:We, at Jet2 (UK's third largest...

  • Lead Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Principal Global Services Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    ResponsibilitiesWhat You'll do:Seeking a Lead Data Engineer with demonstrated ability guiding a team, developing platforms, and aligning strategy with goals.You'll have Opportunit to: - Lead the technical direction and strategy for data engineering projects   - Lead a team of data engineers, providing technical mentorship and guidance   - Establish and...

  • Lead Data Engineer

    5 days ago

    Pune, Maharashtra, India Capco Full time

    About Us"Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors....

  • Lead Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Persistent Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About Position:Were looking for a Lead Technical Engineer who can bridge data engineering depth with FinOps insight. You'll design and implement scalable data solutions across Databricks and Snowflake, drive performance optimization, and ensure that our cloud operations stay cost-efficient and well-governed.Role: Lead Data EngineerLocation: All Persistent...

  • Lead Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Persistent Systems Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    About Position:We're looking for a Lead Technical Engineer who can bridge data engineering depth with FinOps insight. You'll design and implement scalable data solutions across Databricks and Snowflake, drive performance optimization, and ensure that our cloud operations stay cost-efficient and well-governed.Role: Lead Data EngineerLocation: All Persistent...

  • Technical Lead/ Project Lead

    1 week ago

    Pune, Maharashtra, India Cloud Data Vision Full time

    Role Summary:The Technical Product / Project Lead (TPL) will be a key member of K force Consulting Solutions (KCS) - AI CoE team, responsible for managing the execution of AI and automation initiatives from intake through delivery.This individual will coordinate between the U.S. AI CoE leadership, practice SMEs (Cloud, Digital, App Engineering, Data), and...

  • Technical Lead

    2 weeks ago

    Pune, Maharashtra, India Data Axle Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    About Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This center delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by...

  • Data Engineering Lead

    1 week ago

    Pune, Maharashtra, India SiteMinder Full time

    At SiteMinder we believe the individual contributions of our employees are what drive our success. That's why we hire and encourage diverse teams that include and respect a variety of voices, identities, backgrounds, experiences and perspectives. Our diverse and inclusive culture enables our employees to bring their unique selves to work and be proud of...

  • Data Engineering Lead

    2 weeks ago

    Pune, Maharashtra, India SiteMinder Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    At SiteMinder we believe the individual contributions of our employees are what drive our success. That's why we hire and encourage diverse teams that include and respect a variety of voices, identities, backgrounds, experiences and perspectives. Our diverse and inclusive culture enables our employees to bring their unique selves to work and be proud of...