Senior Data Engineer

2 weeks ago

Gurugram, India Steerlean Consulting Services Pvt Ltd. Full time

Job Description :

We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be instrumental in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into data lake and data warehouse.

You will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

Must-Have Skills :

- AWS Glue (Crawler, Data Catalog).

- Python/Pyspark.

- Cloud Skills :

- Experience with Snowflake and its architecture (internal/external tables, stages, masking policies).

- Knowledge of AWS Services like SNS, S3, Lambda, Secret Manager, and Athena.

- Familiarity with Jira, GitHub, and Agile methodology.

Key Responsibilities :

1. ETL Development :


- Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.

- Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.

- Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.

- Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.

- Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.

2. Data Pipeline Management :


- Hands On experience in creating advanced data pipelines using databricks workflows Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.

- Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.

3. Data Analysis and Query Optimization :


- Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.

4. Project Coordination and Continuous Improvement :


- Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.

- Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.

- Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

5. Cloud Platform Expertise :


- Utilize cloud platforms (e.g. , AWS, Azure, or GCP) to design and deploy scalable and reliable SaaS solutions.

- Optimize infrastructure for performance, security, and cost efficiency

(ref:hirist.tech)

  • Senior Data Engineer

    5 hours ago

    Gurugram, India True Tech Professionals Full time

    Senior Data EngineerLocation: Gurgaon (Hybrid Mode)Experience: 5 years and above. Immediate Joiners Preferred (Notice Period: 7 to 10 days)We are seeking a skilled Senior Data Engineer to join our dynamic team. The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL. The...

  • Senior Data Engineer

    15 hours ago

    Gurugram, India True Tech Professionals Full time

    Senior Data EngineerLocation: Gurgaon (Hybrid Mode)Experience: 5 years and above. Immediate Joiners Preferred (Notice Period: 7 to 10 days)We are seeking a skilled Senior Data Engineer to join our dynamic team. The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL. The...

  • Senior Data Engineer

    4 hours ago

    Gurugram, India True Tech Professionals Full time

    Senior Data Engineer Location: Gurgaon (Hybrid Mode) Experience: 5 years and above. Immediate Joiners Preferred (Notice Period: 7 to 10 days) We are seeking a skilled Senior Data Engineer to join our dynamic team. The ideal candidate should have a strong background in data engineering with expertise in Python, Pyspark, AWS Glue, Athena, Redshift, and SQL....

  • Senior Data Engineer

    3 weeks ago

    Chennai/Gurgaon/Gurugram/Delhi NCR/Hyderabad, India Eucloid Data Solutions Pvt Ltd Full time

    **Job Title:** Senior Data Engineer - Databricks Expert**About Eucloid Data Solutions Pvt Ltd**Eucloid Data Solutions Pvt Ltd is a fast-growing AI-consulting firm that enables business growth for its clients through advanced Data Analytics, Machine Learning, and Data engineering. Our team supports various business applications by developing data...

  • Senior Data Engineer

    2 months ago

    Gurugram, India Invokhr Full time

    Position: Senior Data Engineer / Data Engineer Desired Experience: 3-8 years Salary: Best-in-industry You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and implementing big data and analytics solutions. Communication and organisation skills are keys for this position,...

  • Senior Data Engineer

    1 month ago

    Gurugram, India Admin Looks Full time

    Urgently looking for the Senior Data Engineer for onsite at the location of GurgaonJob description:We are seeking a passionate, hands-on and experienced Senior Data Engineer to join our team. You will play a key role in implementing large scale Data platforms with our customers. The candidate will collaborate with internal and customer teams on multiple...

  • Senior Data Engineer

    4 weeks ago

    Chennai/Gurgaon/Gurugram/Delhi NCR/Hyderabad, India Eucloid Data Solutions Pvt Ltd Full time

    About the Role:Eucloid Data Solutions Pvt Ltd is seeking a highly skilled Senior Data Engineer to join our Data Platform team. The successful candidate will have hands-on expertise in Databricks and will be responsible for designing and building data infrastructure to support various business applications.Key Responsibilities:Design, implementation, and...

  • Senior Data Engineer

    4 weeks ago

    Chennai/Gurgaon/Gurugram/Delhi NCR/Hyderabad, India Eucloid Data Solutions Pvt Ltd Full time

    About Eucloid Data SolutionsEucloid Data Solutions is a fast-growing AI-consulting firm that enables business growth for its clients through advanced Data Analytics, Machine Learning and Data engineering. We offer bespoke, full stack and data-driven approach to solve highly complex and large-scale business problems for some of the biggest Fortune 500...

  • Senior Data Engineer

    1 month ago

    gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer IIExperience: Must have 4+ years of experienceAbout Role:We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences.About us:Nurtured from the seed of a single great idea - to empower the traveler - MakeMyTrip went...

  • Senior Data Engineer

    1 month ago

    Gurgaon/Gurugram/Noida, India Om Careers Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team. As a key member of our Data Engineering team, you will be responsible for designing, developing, and implementing data engineering solutions to meet the demands of our business requirements.Key Responsibilities:Develop and implement data collection systems...

  • Senior Data Engineer

    3 months ago

    Gurugram, India Invokhr Full time

    Desired Experience: 3-8 years Salary: Best-in-industry Location: Gurgaon ( 5 days onsite) Overview: You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and implementing big data and analytics solutions. Communication and organisation skills are keys for this position,...

  • Senior Data Engineer

    2 weeks ago

    gurugram, India NCR Voyix Full time

    About NCR Voyix CorporationNCR Voyix (NYSE: VYX) is a global technology company leading how the world connects, interacts and transacts with business. NCR Voyix’s assisted and self-service solutions and comprehensive support services address the needs of retail & restaurants organizations in more than 100 countries. NCR Voyix (www.ncr.com) is headquartered...

  • Senior Data Engineer

    2 weeks ago

    Gurugram, India Partech Full time

    Hi there! Have you noticed that the restaurant industry is changing?We are PAR and our purpose is:To deliver solutions that connect people to the restaurants, meals, and moments they love.At PAR Technology, our relentless drive for innovation and unwavering commitment to customer success are at the heart of everything we do. We lead the restaurant and retail...

  • Senior Data Engineer

    2 months ago

    gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer IIExperience: Must have 4+ years of experienceAbout Role:We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences.About us:Nurtured from the seed of a single great idea - to empower the traveler - MakeMyTrip went...

  • Senior Data Engineer

    2 months ago

    gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer II Experience: Must have 4+ years of experience About Role: We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences. About us: Nurtured from the seed of a single great idea - to empower the traveler -...

  • Senior Data Engineer

    2 months ago

    gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer IIExperience: Must have 4+ years of experienceAbout Role:We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences.About us:Nurtured from the seed of a single great idea - to empower the traveler - MakeMyTrip went...

  • Senior Data Engineer

    2 months ago

    Gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer IIExperience: Must have 4+ years of experienceAbout Role:We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences.About us:Nurtured from the seed of a single great idea - to empower the traveler - MakeMyTrip went...

  • Senior Data Engineer

    2 months ago

    Gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer II Experience: Must have 4+ years of experience About Role: We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences. About us: Nurtured from the seed of a single great idea - to empower the traveler -...

  • Senior Data Engineer

    2 months ago

    Gurugram, India MakeMyTrip Full time

    Position: Senior Data Engineer IIExperience: Must have 4+ years of experienceAbout Role:We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences.About us:Nurtured from the seed of a single great idea - to empower the traveler - MakeMyTrip went...

  • Eucloid - Senior Data Engineer - Azure Databricks

    6 months ago

    Chennai/Gurgaon/Gurugram, India Eucloid Data Solutions Pvt Ltd Full time

    Job Role : Sr. Data Engineer (Databricks) About Eucloid :Eucloid Data Solutions is a fast-growing AI-consulting firm that enables business growth for its clients through advanced Data Analytics, Machine Learning and Data engineering. Eucloid offers bespoke, full stackand datadriven approach to solve highly complex and large-scale business problems for some...