Data Engineer for Scalable Data Pipelines

1 week ago


Gurgaon, Haryana, India Steerlean Consulting Services Pvt Ltd. Full time
Job Description
We are seeking a highly skilled Data Engineer to join our team at Steerlean Consulting Services Pvt Ltd. The successful candidate will be responsible for designing, developing, and maintaining scalable data pipelines using Databricks PySpark.

The ideal candidate will have experience with AWS Glue, Python/Pyspark, and cloud skills such as Snowflake and its architecture, AWS Services like SNS, S3, Lambda, Secret Manager, and Athena. Familiarity with Jira, GitHub, and Agile methodology is also required.

Key Responsibilities:
1. ETL Development
• Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
• Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
• Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.

2. Data Pipeline Management
• Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
• Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.

3. Data Analysis and Query Optimization
• Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.

4. Project Coordination and Continuous Improvement
• Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
• Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.

Estimated Salary: $120,000 - $150,000 per year.
  • Data Engineer

    1 month ago


    Gurgaon, Haryana, India MINFY TECHNOLOGIES PRIVATE LIMITED, Full time

    Job Title : Data Engineer - Cloud Data Pipeline SpecialistWe are seeking an experienced Data Engineer to join our team at Minfy Technologies Private Limited. The estimated salary for this role is ₹1,500,000 - ₹2,500,000 per annum, depending on experience. Company OverviewMinfy Technologies Private Limited is a leading technology company that specializes...


  • Gurgaon, Haryana, India MINFY TECHNOLOGIES PRIVATE LIMITED, Full time

    Senior Data Engineer PositionWe are looking for a skilled Senior Data Engineer to join our team at Minfy Technologies. As a Senior Data Engineer, you will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure to support data-driven initiatives.About the Role:Lead the design and implementation of robust data...


  • Gurgaon, Haryana, India True Tech Professionals Full time

    Job DescriptionWe are seeking a skilled Senior Data Engineer to join our dynamic team at True Tech Professionals. The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL.About the RoleThe successful candidate will play a key role in designing, developing, and maintaining...


  • Gurgaon, Haryana, India AXA Group Full time

    Experience the excitement of transforming data into actionable insights as a Senior Data Pipeline Engineer at AXA XL.About UsAXA XL is a leading provider of specialty risk solutions, serving mid-sized companies and multinationals with innovative and flexible approaches to risk management. We're known for combining a comprehensive capital platform,...


  • Gurgaon, Haryana, India Rapture Staffing Full time

    About the RoleAt Rapture Staffing, we are seeking an experienced Senior Data Engineer to join our team. This role will be responsible for designing, implementing, and optimizing scalable and efficient data architectures and pipelines to support various data use cases.


  • Gurgaon, Haryana, India Axis Max Life Insurance Limited Full time

    We are seeking an experienced Big Data Engineer to join our team at Axis Max Life Insurance Limited.The estimated salary for this role is approximately $180,000 per annum, depending on the location and candidate qualifications.About the Role:This senior position involves designing, implementing, and maintaining large-scale data pipelines and infrastructure...

  • Cloud Data Engineer

    2 weeks ago


    Gurgaon, Haryana, India Tredence Analytics Solutions Private Limited Full time

    Job Title: Cloud Data Engineer - Scalable SolutionsWe are seeking an experienced Senior GCP Data Engineer to join our team. The ideal candidate will have a strong background in designing, implementing, and architecting data solutions on Google Cloud Platform (GCP). With minimum 5 years of experience in IT or professional services, you will be responsible for...


  • Gurgaon, Haryana, India Steerlean Consulting Services Pvt Ltd. Full time

    Job Summary: We are looking for a highly skilled Scalable Data Solutions Architect to join our team at Steerlean Consulting Services Pvt Ltd. The successful candidate will design, build, and maintain scalable data pipelines and systems to support analytics and business intelligence.In this role, you will be responsible for optimizing database performance and...


  • Gurgaon, Haryana, India Newnovation Full time

    Job Title: Scalable Data Solutions ExpertWe are seeking a skilled and experienced Data Engineer to join our team at Newnovation in Gurugram. The ideal candidate will have a strong background in data processing, cloud-based solutions, and big data technologies.About the Role:Develop, maintain, and optimize scalable ETL processes using PySpark and Python.Write...


  • Gurgaon, Haryana, India exl Full time

    About the RoleWe are seeking a highly skilled Data Engineering Manager to lead our team in designing and building scalable data systems. As a key member of our engineering team, you will be responsible for collaborating with cross-functional teams to develop and maintain our operations data mart, ensuring a stable reporting...


  • Gurgaon, Haryana, India Canara HSBC Life Insurance Full time

    This is a leadership role within the engineering team at Canara HSBC Life Insurance, focused on end-to-end data lifecycle management.Key Responsibilities:Design and develop cloud-based data pipelines to process high volumes of data from ingestion to consumption.Collaborate with internal and vendor-side engineering teams to ensure seamless data movement...

  • Senior ETL Developer

    2 weeks ago


    Gurgaon, Haryana, India Falcon Full time

    Salary Estimate: ₹1200000 - ₹1800000 per annumJob Description:Design and develop scalable data pipelines using cloud-based services.Maintain and optimize existing data pipeline architecture.Formulate complex SQL queries for big data processing.Collaborate with product and engineering teams to design and develop a platform for data modeling and machine...

  • Data Pipeline Analyst

    1 month ago


    Gurgaon, Haryana, India American Express Full time

    **Job Overview**At American Express, we are seeking a highly skilled Data Pipeline Analyst to join our team. As a key member of our data analytics team, you will be responsible for designing, building, and maintaining complex data pipelines to support our business operations.**Key Responsibilities**Participate in the full lifecycle of data pipeline...

  • Data Engineering Lead

    2 weeks ago


    Gurgaon, Haryana, India Consultancy Full time

    Job Title: Data Engineering LeadThe Consultancy is seeking an experienced Data Engineer to contribute to our expanding team. As a Data Engineering Lead, you will be responsible for developing scalable data pipelines, ensuring high data quality, and enabling data-driven decision-making by building optimized solutions on cloud platforms like Azure.Key...


  • Gurgaon, Haryana, India TalentXO Full time

    Data Engineering SpecialistWe are seeking a highly skilled Data Engineering Specialist to join our client's team, who offers online cohort-based courses for architects, engineers, and designers in the AEC industry.The ideal candidate will have experience in designing, implementing, and optimizing scalable and efficient data architectures and pipelines to...


  • Gurgaon, Haryana, India WEN Full time

    At WEN, we are looking for a seasoned Senior Data Pipeline Specialist to lead our data engineering efforts. As a key member of our team, you will be responsible for designing and implementing robust data pipelines that drive business growth and inform strategic decision-making.The ideal candidate will have extensive experience in data warehousing, cloud...


  • Gurgaon, Haryana, India Steerlean Consulting Services Pvt Ltd. Full time

    Job Title: Data Engineering ExpertSteerlean Consulting Services Pvt Ltd. is seeking a highly skilled Data Engineer to join our team. The successful candidate will design, build, and maintain scalable data pipelines and systems to support analytics and business intelligence.This role requires hands-on experience in big data tools, cloud platforms, and...

  • Data Architect

    3 weeks ago


    Gurgaon, Haryana, India Impetus Full time

    Job SummaryWe are seeking a skilled Data Architect - Big Data Engineer to join our team at Impetus. As a key member of our data engineering team, you will design, develop, and implement efficient ETL pipelines using big data technologies.About the RoleDevelop scalable and secure data pipelines using Hadoop, Hive, and PySpark.Collaborate with cross-functional...

  • Cloud Data Engineer

    3 weeks ago


    Gurgaon, Haryana, India Cognitio Analytics Inc Full time

    Job SummaryCognitio Analytics Inc is seeking an experienced Cloud Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing and implementing scalable cloud-based data solutions using Microsoft Azure.About the RoleWe are looking for a highly skilled professional with expertise in Azure services and...


  • Gurgaon, Haryana, India ProcDNA Full time

    About ProcDNAWe are a global consulting firm that fuses design thinking with cutting-edge technology to create innovative Commercial Analytics and Technology solutions for our clients. With a passionate team of 200+ across 6 offices, we are committed to growth and learning together.Job Description:We are seeking an experienced Data Engineer to build and...