Lead Data Engineer

2 weeks ago

solapur, India ITC Infotech Full time

Lead Data EngineerLocation: Banglore/Pune (Hybrid)Mode: HybridShift Timing: 2 PM to 11 PMExperience: 7-12 years. 3+ years in cloud data platforms and Databricks.Purpose:We are seeking a hands-on Technical Lead to drive the ingestion of high-volume mainframe RPC data into Databricks, enabling scalable machine learning workflows. This role is critical to building a robust data foundation for training thousands of AI models that detect anomalous behavior across applications, services, and functions.Key Responsibilities- Mainframe Data Ingestion:- Design and implement scalable pipelines to extract, parse, and ingest RPC logs and technical attributes from mainframe systems into Delta Lake on Databricks.- ML-Ready Data Engineering:- Transform and structure data for time-series modelling and anomaly detection across thousands of models.- ML Workflow Integration:- Collaborate with ML engineers to ensure data pipelines support SARIMA, ANN, and other model types; enable automated retraining and scoring.- Performance Optimization:- Tune Spark jobs, Delta Lake storage, and cluster configurations for billions of records and real-time aggregation.- FinOps & Cost Control:- Monitor and optimize Databricks resource usage; implement auto-scaling and cost-aware job scheduling.- Monitoring & Alerting:- Integrate Databricks-native alerting for pipeline health, data anomalies, and job failures.Required Skills- Strong hands-on experience with Databricks, Apache Spark, Delta Lake, and MLflow- Proven expertise in mainframe data integration (e.g., SMF, RPC logs, VSAM, DB2)- Strong Python and PySpark programming skillsPreferred Skills- Databricks experience with CI/CD tools GitHub Actions- Knowledge of FinOps principles for cloud cost optimization

  • Senior Data Engineering Lead

    1 week ago

    solapur, India beBeeData Full time

    As a seasoned data engineering professional, this role offers the opportunity to lead and architect our organization's data strategy.Key ResponsibilitiesThe successful candidate will have at least 6 years of experience in designing and implementing data engineering solutions.A strong background in CI-CD, PySpark, SQL, ADF, ADLS & ADB is essential for success...

  • Sustainability Data Engineering Lead

    2 weeks ago

    solapur, India beBeeDataEngineer Full time

    Job Title: Sustainability Data Engineer LeadSustainability Initiative SupportThe successful candidate will be responsible for building data pipelines and models to support product-level carbon footprint analysis using Databricks on AWS.Main Responsibilities:Develop and optimize ETL/ELT pipelines on Databricks using PySpark and SQL to support carbon footprint...

  • Cloud Native Data Engineering Lead

    5 days ago

    solapur, India beBeeDataEngineering Full time

    Senior Data Engineering ManagerWe are seeking a highly skilled Senior Data Engineering Manager to lead the development of our cloud-native data warehouse on Amazon Web Services (AWS).Develop a modern data warehousing architecture using AWS services, ensuring scalability, security, and efficiency.Leverage your technical expertise to manage and mentor a team...

  • Lead Data Architect

    2 weeks ago

    solapur, India beBeeDataEngineer Full time

    Key Data Engineer RoleWe are seeking a seasoned professional to lead our data engineering initiatives.Create and optimize complex data pipelines and workflows, driving efficiency and scalability in cloud-based environments.Develop high-quality Python scripts for data processing, emphasizing reusability and maintainability.Design and implement robust SQL...

  • Senior Data Engineering Project Lead

    2 weeks ago

    solapur, India beBeeDataEngineeringManager Full time

    Job OverviewWe are seeking an experienced Technical Project Manager to lead and deliver projects in data engineering and analytics.This role involves managing cross-functional teams to execute data platform, pipeline, and analytics initiatives, ensuring alignment with business goals and timely delivery.Key ResponsibilitiesProject Planning: Plan, execute, and...

  • Leading Cloud Data Architect

    2 weeks ago

    solapur, India beBeeDataLead Full time

    Senior Data Architecture LeadWe are seeking a Senior Data Architecture Lead to lead solution design and delivery execution for data, analytics, and AI programs across Clinical, Commercial, and Medical Affairs domains.This individual will be responsible for designing end-to-end data platform architectures, building scalable solutions on leading cloud...

  • Lead Data Pipeline Developer

    1 week ago

    solapur, India beBeeDataEngineer Full time

    Data Engineering involves designing and implementing scalable data pipelines using technologies like Apache Airflow, Spark, AWS services, and Python.Key ResponsibilitiesDesign and implement robust data pipelines to support business growth.Develop and orchestrate workflows using Apache Airflow and Spark for efficient data processing.Leverage cloud-based...

  • Principal Data Engineer

    2 weeks ago

    Solapur, India Nexuspoint Consultant Full time

    Job Title: Principal Data Engineer Contract Period: more than 6 months Experience: 6+ Working hours: 9:30am to 6:30pm IST About the Role We are looking for an accomplished Principal Data Engineer to lead the ideation, architecture, design and development of our next-generation enterprise data platform. This role requires a visionary technologist with deep...

  • Advanced Data Engineer

    2 weeks ago

    solapur, India beBeeData Full time

    OverviewWe are seeking a highly skilled professional to join our team as an ETL Tester.Your RoleYou will be responsible for designing, developing and executing ETL test cases, scenarios and scripts for validation and regression testing.Validating data extraction, transformation and loading processes for accuracy and completeness is also a key aspect of this...

  • Leading Data Architect

    1 week ago

    solapur, India beBeeDataEngineer Full time

    Data Engineering Expert We are seeking a seasoned professional to fill a pivotal role in our organization. About the Position: Design and build high-performance data processing systems on Google Cloud Platform (GCP) Collaborate with cross-functional teams to gather requirements and implement innovative data solutions Partner with data scientists and...