Lead Data Engineer
2 hours ago
Lead Data Engineer
Location: Banglore/Pune (Hybrid)
Mode: Hybrid
Shift Timing: 2 PM to 11 PM
Experience: 7-12 years. 3+ years in cloud data platforms and Databricks.
Purpose:
We are seeking a hands-on Technical Lead to drive the ingestion of high-volume mainframe RPC data into Databricks, enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.
Key Responsibilities
- Mainframe Data Ingestion:
- Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.
- ML-Ready Data Engineering:
- Transform and structure data for time-series modelling and anomaly detection across thousands of models.
- ML Workflow Integration:
- Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types; enable automated retraining and scoring.
- Performance Optimization:
- Tune Spark jobs, Delta Lake storage, and cluster configurations for billions of records and real-time aggregation.
- FinOps & Cost Control:
- Monitor and optimize Databricks resource usage; implement auto-scaling and cost-aware job scheduling.
- Monitoring & Alerting:
- Integrate Databricks-native alerting for pipeline health, data anomalies, and job failures.
Required Skills
- Strong hands-on experience with Databricks, Apache Spark, Delta Lake, and MLflow
- Proven expertise in mainframe data integration (e.g., SMF, RPC logs, VSAM, DB2)
- Strong Python and PySpark programming skills
Preferred Skills
- Databricks experience with CI/CD tools GitHub Actions
- Knowledge of FinOps principles for cloud cost optimization
-
Lead Data Engineer
2 hours ago
Pune, Maharashtra, India, Maharashtra Searce Inc Full timeWho we are?Searce means ‘a fine sieve’ & indicates ‘to refine, to analyze, to improve’. It signifies our way of working: To improve to the finest degree of excellence, ‘solving for better’ every time. Searcians are passionate improvers & solvers who love to question the status quo.The primary purpose of all of us, at Searce, is driving...
-
Lead Data Engineer
2 hours ago
Pune, Maharashtra, India, Maharashtra EXL Full timeAbout the RoleWe are seeking a highly skilled Lead Data Engineer to drive end-to-end data engineering initiatives, lead cross-functional teams, and deliver scalable, cloud-based data solutions. The ideal candidate will bring deep technical expertise, strong leadership, and the ability to collaborate effectively with global stakeholders in a fast-paced...
-
Data Privacy Consultant
2 hours ago
Pune, Maharashtra, India, Maharashtra Data > Nuance. Full timeAbout Data>Nuance Data>Nuance is a global privacy, data protection, and AI governance consultancy, trusted by 1,000+ organizations worldwide.We operate across Barcelona, Bangalore, and Dubai (expanding soon), with a dedicated team of data protection, governance, and regulatory specialists.We specialize in:Outsourced DPO ServicesData Protection Consultancy &...
-
Data and Analytics Engineering Lead
2 hours ago
Pune, Maharashtra, India, Maharashtra Orbia Pune Full timeWe are currently looking for Data and Analytics Engineering Lead to join a fast growing product based Organisation in Pune.If you are someone who can develop a comprehensive technical plan for the migration, including data ingestion, transformation, storage, and access control in Azure's Data Factory and data lake! We'd love to hear from you.Role: Data and...
-
Google Cloud Data engineering Practice Lead
2 hours ago
Pune, Maharashtra, India, Maharashtra Evonence Full timeCompany DescriptionEvonence is a Google Cloud partner company founded in 2014 and located in Pune. As one of the fastest-growing partners in the Google Cloud ecosystem, we provide Google Cloud solutions specifically tailored to the mid-market across North America. With a customer base of over 1000 clients, we have deep technical expertise in Google...
-
Lead
2 hours ago
Pune, Maharashtra, India, Maharashtra Yotta Data Services Private Limited Full timeYotta Data Services | Powering Digital Transformation with Scalable Cloud, Colocation, and Managed Services. Yotta Data Services offers a comprehensive suite of cloud, data center, and managed services designed to accelerate digital transformation for businesses of all sizes. With state-of-the-art infrastructure, cutting-edge AI capabilities, and a...
-
Lead Data Engineer
6 days ago
Pune, Maharashtra, India EXL Talent Acquisition Team Full time ₹ 15,00,000 - ₹ 25,00,000 per yearLead Data Engineer Job DescriptionRole Overview:We are looking for a skilled Lead Data Engineer to design, build, and manage scalable data pipelines and architectures. You will lead a team of data engineers, collaborate with cross-functional stakeholders, and drive best practices in data engineering to support data-driven decision-making.Key...
-
Data Practice Lead
2 hours ago
Pune, Maharashtra, India, Maharashtra Opus Technologies Full timeJob Title: Data Practice LeadLocation: Pune (Preferred) / BengaluruExperience: 15+ YearsEmployment Type: Full-TimePrimary Skills: Data Engineering, Data Platforms, Competency Building, Practice Growth, Consulting/Advisory, Delivery/Operational Excellence, CoE, Team Building, Databricks, SnowflakeJob Scope: We are seeking a dynamic Data Engineering...
-
Lead Data Engineer
2 hours ago
Pune, Maharashtra, India, Maharashtra Bajaj Technology Services Full timeExperience: 9-12 YearsDomain: eCommerce / Financial ReconciliationBrief JD:We are seeking a Sr. Data Engineer own and operate a large-scale data processing and reconciliation platform for an eCommerce enterprise. The role involves execution, support and enhancement of real-time data ingestion and batch processing ETL dataflows (SQL, Python, PySpark, Hive),...
-
Lead Data Scientist
2 hours ago
Pune, Maharashtra, India, Maharashtra Ishan Technologies Full timeJob Title: Lead Data ScientistLocation: PuneSalary: Upto 22 LPAJob Summary:We are seeking a highly experienced and visionary Lead Data Scientist to spearhead our data-driven initiatives and guide a team of data professionals in developing scalable, impactful solutions. The ideal candidate will have deep expertise in machine learning, predictive analytics,...