Lead Data Engineer

2 hours ago


Pune Maharashtra India, Maharashtra ITC Infotech Full time

Lead Data Engineer


Location: Banglore/Pune (Hybrid)

Mode: Hybrid

Shift Timing: 2 PM to 11 PM


Experience: 7-12 years. 3+ years in cloud data platforms and Databricks.

Purpose:

We are seeking a hands-on Technical Lead to drive the ingestion of high-volume mainframe RPC data into Databricks, enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.


Key Responsibilities

  • Mainframe Data Ingestion:
  • Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.
  • ML-Ready Data Engineering:
  • Transform and structure data for time-series modelling and anomaly detection across thousands of models.
  • ML Workflow Integration:
  • Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types; enable automated retraining and scoring.
  • Performance Optimization:
  • Tune Spark jobs, Delta Lake storage, and cluster configurations for billions of records and real-time aggregation.
  • FinOps & Cost Control:
  • Monitor and optimize Databricks resource usage; implement auto-scaling and cost-aware job scheduling.
  • Monitoring & Alerting:
  • Integrate Databricks-native alerting for pipeline health, data anomalies, and job failures.

Required Skills

  • Strong hands-on experience with Databricks, Apache Spark, Delta Lake, and MLflow
  • Proven expertise in mainframe data integration (e.g., SMF, RPC logs, VSAM, DB2)
  • Strong Python and PySpark programming skills

Preferred Skills

  • Databricks experience with CI/CD tools GitHub Actions
  • Knowledge of FinOps principles for cloud cost optimization


  • Lead Data Engineer

    2 hours ago


    Pune, Maharashtra, India, Maharashtra Searce Inc Full time

    Who we are?Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’. It signifies our way of working: To improve to the finest degree of excellence, ‘solving for better’ every time. Searcians are passionate improvers & solvers who love to question the status quo.The primary purpose of all of us, at Searce, is driving...

  • Lead Data Engineer

    2 hours ago


    Pune, Maharashtra, India, Maharashtra EXL Full time

    About the RoleWe are seeking a highly skilled Lead Data Engineer to drive end-to-end data engineering initiatives, lead cross-functional teams, and deliver scalable, cloud-based data solutions. The ideal candidate will bring deep technical expertise, strong leadership, and the ability to collaborate effectively with global stakeholders in a fast-paced...


  • Pune, Maharashtra, India, Maharashtra Data > Nuance. Full time

    About Data>Nuance Data>Nuance is a global privacy, data protection, and AI governance consultancy, trusted by 1,000+ organizations worldwide.We operate across Barcelona, Bangalore, and Dubai (expanding soon), with a dedicated team of data protection, governance, and regulatory specialists.We specialize in:Outsourced DPO ServicesData Protection Consultancy &...


  • Pune, Maharashtra, India, Maharashtra Orbia Pune Full time

    We are currently looking for Data and Analytics Engineering Lead to join a fast growing product based Organisation in Pune.If you are someone who can develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake! We'd love to hear from you.Role: Data and...


  • Pune, Maharashtra, India, Maharashtra Evonence Full time

    Company DescriptionEvonence is a Google Cloud partner company founded in 2014 and located in Pune. As one of the fastest-growing partners in the Google Cloud ecosystem, we provide Google Cloud solutions specifically tailored to the mid-market across North America. With a customer base of over 1000 clients, we have deep technical expertise in Google...

  • Lead

    2 hours ago


    Pune, Maharashtra, India, Maharashtra Yotta Data Services Private Limited Full time

    Yotta Data Services | Powering Digital Transformation with Scalable Cloud, Colocation, and Managed Services. Yotta Data Services offers a comprehensive suite of cloud, data center, and managed services designed to accelerate digital transformation for businesses of all sizes. With state-of-the-art infrastructure, cutting-edge AI capabilities, and a...

  • Lead Data Engineer

    6 days ago


    Pune, Maharashtra, India EXL Talent Acquisition Team Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Lead Data Engineer Job DescriptionRole Overview:We are looking for a skilled Lead Data Engineer to design, build, and manage scalable data pipelines and architectures. You will lead a team of data engineers, collaborate with cross-functional stakeholders, and drive best practices in data engineering to support data-driven decision-making.Key...

  • Data Practice Lead

    2 hours ago


    Pune, Maharashtra, India, Maharashtra Opus Technologies Full time

    Job Title: Data Practice LeadLocation: Pune (Preferred) / BengaluruExperience: 15+ YearsEmployment Type: Full-TimePrimary Skills: Data Engineering, Data Platforms, Competency Building, Practice Growth, Consulting/Advisory, Delivery/Operational Excellence, CoE, Team Building, Databricks, SnowflakeJob Scope: We are seeking a dynamic Data Engineering...

  • Lead Data Engineer

    2 hours ago


    Pune, Maharashtra, India, Maharashtra Bajaj Technology Services Full time

    Experience: 9-12 YearsDomain: eCommerce / Financial ReconciliationBrief JD:We are seeking a Sr. Data Engineer own and operate a large-scale data processing and reconciliation platform for an eCommerce enterprise. The role involves execution, support and enhancement of real-time data ingestion and batch processing ETL dataflows (SQL, Python, PySpark, Hive),...

  • Lead Data Scientist

    2 hours ago


    Pune, Maharashtra, India, Maharashtra Ishan Technologies Full time

    Job Title: Lead Data ScientistLocation: PuneSalary: Upto 22 LPAJob Summary:We are seeking a highly experienced and visionary Lead Data Scientist to spearhead our data-driven initiatives and guide a team of data professionals in developing scalable, impactful solutions. The ideal candidate will have deep expertise in machine learning, predictive analytics,...