Senior Data Engineer

6 days ago

Bangalore, India Careerhaat Full time

Job Location : - Bengaluru (hybrid)

Permanent position with company payroll

Job Description :

We are looking for a highly skilled Databricks PySpark Developer to join our data platform implementation team. In this role, you will be instrumental in designing, developing, and maintaining ETL processes to ensure efficient extraction, transformation, and loading of data from various sources into data lake and data warehouse.


You will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

Key Responsibilities :

1. ETL Development : Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse. Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes. Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes. Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers. Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.

2. Data Pipeline Management : Hands On experience in creating advanced data pipelines using databricks workflows Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle. Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.

3. Data Analysis and Query Optimization : Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.

4. Project Coordination and Continuous Improvement : Participate in project planning and coordination activities to ensure timely delivery of ETL solutions. Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations. Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Qualifications :

- Bachelor's degree in Computer Science, Engineering, or a related field.


- Minimum of 3 years of experience on Databricks PySpark development.


- Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.


- In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.


- Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.


- Proven experience in working with large-scale data warehousing and ETL frameworks.


- Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.


- Excellent communication and collaboration skills, with the ability to work effectively in a team Qualifications :


- Experience with cloud platforms with a preference for AWS. Experience with data platform tools such as DataBricks, Snowflake, and Tableau.


- Demonstrated ability to implement best practices for ETL processes and data management.


- Strong understanding of data governance and data quality principles.


- Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

(ref:hirist.tech)

  • Hevo Data

    6 days ago

    Bangalore/Bangalore, India HEVO DATA Full time

    This role provides a great opportunity to own and build highly scalable and fault-tolerant platform services that will help support thousands of customers' data pipelines and load data reliability to the different cloud warehouses. These services are the core of the Hevo platform that helps provide building no code-low pipeline. Also, makes a huge...

  • Senior data engineer

    3 weeks ago

    Bangalore, India Rakuten Symphony Full time

    Job Title: Senior Data Engineer Location: Bangalore, India Company: Rakuten Symphony, India Overview Rakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise in...

  • Data engineer

    3 months ago

    Bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost...

  • Data Engineer

    3 months ago

    Bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and...

  • Senior Data Engineer

    1 month ago

    Bangalore, India Rakuten Symphony Full time

    Job Title: Senior Data Engineer Location: Bangalore, India Company: Rakuten Symphony, India Overview Rakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise...

  • Senior Data Engineer

    1 month ago

    Bangalore, India Rakuten Symphony Full time

    Job Title: Senior Data Engineer Location: Bangalore, India Company: Rakuten Symphony, India Overview Rakuten Symphony is seeking an experienced and strategic Head/Senior Data Engineer to lead our data engineering initiatives and transform mobile connectivity through innovative data solutions. This senior leadership role requires deep expertise...

  • Data Engineer

    2 months ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices.Deliver assigned projects within budgeted time, cost and agreed...

  • Senior Data Engineer

    1 month ago

    Bangalore, India ThoughtFocus Full time

    Job Description: Senior Data Engineer Position: Senior Data Engineer Experience: 8-13 Years Location: Bangalore Employment Type: Full-Time Must have experience in Capital Market/Investment banking/BFSI domain. ONLY IMMEDIATE JOINERS (0-15 DAYS) Summary: We are looking for a highly skilled Senior Data Engineer with 8-13 years of...

  • Senior Data Engineer

    2 months ago

    Bangalore, India ThoughtFocus Full time

    Job Description: Senior Data Engineer Position: Senior Data Engineer Experience: 8-10 Years Location: Bangalore Employment Type: Full-Time Must have experience in Capital Markets, Investment banking domain. --- Summary: We are looking for a highly skilled Senior Data Engineer with 8-10 years of experience in SQL, Azure Data Factory (ADF),...

  • Senior Data Engineer

    2 months ago

    bangalore, India ThoughtFocus Full time

    Job Description: Senior Data EngineerPosition: Senior Data EngineerExperience: 8-10 YearsLocation: BangaloreEmployment Type: Full-TimeMust have experience in Capital Markets, Investment banking domain.---Summary:We are looking for a highly skilled Senior Data Engineer with 8-10 years of experience in SQL, Azure Data Factory (ADF), SQL Server Integration...

  • Senior Data Engineer

    2 months ago

    bangalore, India ThoughtFocus Full time

    Job Description: Senior Data Engineer Position: Senior Data Engineer Experience: 8-10 Years Location: Bangalore Employment Type: Full-Time Must have experience in Capital Markets, Investment banking domain. --- Summary: We are looking for a highly skilled Senior Data Engineer with 8-10 years of experience in SQL, Azure Data Factory (ADF), SQL Server...

  • Data Engineer

    3 months ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders. Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and...

  • Data Engineer

    3 months ago

    bangalore, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and agreed...

  • Senior Data Engineer

    2 months ago

    Bangalore, India Capco Full time

    Greetings from Capco... We are hiring for Senior Data Engineer for Bangalore Location. Qualifications: Proven 5-8 years of experience as a Senior Data Engineer or similar role. • Experience with big data tools: Hadoop, Spark, Scala, Hive, etc. • Expert level SQL skills for data manipulation (DML) and validation (DB2). • Experience with...

  • Senior Data Engineer

    2 months ago

    bangalore, India Capco Full time

    Greetings from Capco...We are hiring for Senior Data Engineer for Bangalore Location.Qualifications:Proven 5-8 years of experience as a Senior Data Engineer or similar role.• Experience with big data tools: Hadoop, Spark, Scala, Hive, etc.• Expert level SQL skills for data manipulation (DML) and validation (DB2).• Experience with data pipeline and...

  • Senior data engineer

    4 weeks ago

    Bangalore, India Publicis Sapient Full time

    Senior Associate Data Engineering Level 2 The Senior Associate People Senior Associate L2 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing package...

  • Senior Data Engineer

    1 month ago

    Bangalore Urban, India ThoughtFocus Full time

    Job Description: Senior Data Engineer Position: Senior Data Engineer Experience: 8-13 Years Location: Bangalore Employment Type: Full-Time Must have experience in Capital Market/Investment banking/BFSI domain. ONLY IMMEDIATE JOINERS (0-15 DAYS) Summary: We are looking for a highly skilled Senior Data Engineer with 8-13 years of experience in SQL,...

  • Senior Data Engineer

    1 month ago

    Bangalore Urban, India ThoughtFocus Full time

    Job Description: Senior Data EngineerPosition: Senior Data Engineer Experience: 8-13 Years Location: Bangalore Employment Type: Full-TimeMust have experience in Capital Market/Investment banking/BFSI domain.ONLY IMMEDIATE JOINERS (0-15 DAYS) Summary:We are looking for a highly skilled Senior Data Engineer with 8-13 years of experience in SQL, Python, Pyspark...

  • Senior Data Engineer

    4 weeks ago

    Bangalore, India Publicis Sapient Full time

    Senior Associate Data Engineering Level 2 The Senior Associate People Senior Associate L2 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing...

  • Senior Data Engineer

    1 month ago

    Bangalore, India Publicis Sapient Full time

    Senior Associate Data Engineering Level 2 The Senior Associate People Senior Associate L2 in Data Engineering, you will translate client requirements into technical design, and implement components for data engineering solution. Utilize deep understanding of data integration and big data design principles in creating custom solutions or implementing...