Lead Data Engineer

1 month ago


Bengaluru, India SourceBae Full time

Lead Data Engineer-MS Fabric.

Exp : 10+ years.

Type : Contract(remote).

Shift Time : 3PM to 12AM IST.

Description :

We are looking for a Lead Data Engineer with extensive experience in developing ETL processes using PySpark Notebooks and Microsoft Fabric and supporting existing legacy SQL Server environments.

The ideal candidate will possess a strong background in Spark-based development, demonstrate a high proficiency in SQL, and be comfortable working independently, collaboratively within a team, or leading other developers when required.

- Design, develop, and maintain ETL pipelines using PySpark Notebooks and Microsoft Fabric.

- Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver efficient data solutions.

- Migrate and integrate data from legacy SQL Server environments into modern data platforms.

- Optimize data pipelines and workflows for scalability, efficiency, and reliability.

- Provide technical leadership and mentorship to junior developers and other team members.

- Troubleshoot and resolve complex data engineering issues related to performance, data quality, and system scalability.

- Develop, maintain, and enforce data engineering best practices, coding standards, and documentation.

- Conduct code reviews and provide constructive feedback to improve team productivity and code quality.

- Support data-driven decision-making processes by ensuring data integrity, availability, and consistency across different platforms.

Requirements :

- Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related field.

- Experience with Microsoft Fabric or similar cloud-based data integration platforms is a must.

- 10+ years of experience in data engineering, with a strong focus on ETL development using PySpark or other Spark-based tools.

- Proficiency in SQL with extensive experience in complex queries, performance tuning, and data modeling.

- Strong knowledge of data warehousing concepts, ETL frameworks, and big data processing.

- Familiarity with other data processing technologies (e.g, Hadoop, Hive, Kafka) is an advantage.

- Experience working with both structured and unstructured data sources.

- Excellent problem-solving skills and the ability to troubleshoot complex data engineering issues.

- Proven ability to work independently, as part of a team, and in leadership roles.

- Strong communication skills with the ability to translate complex technical concepts into business terms.

Mandate skills :

- Experience with Data lake, Data warehouse, Delta lake.

- Experience with Azure Data Services, including Azure Data Factory, Azure Synapse, or similar tools.

- Knowledge of scripting languages (e.g, Python, Scala) for data manipulation and automation.

- Familiarity with DevOps practices, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus.

(ref:hirist.tech)

  • Bengaluru, Karnataka, India HEVO DATA Full time

    Scale Data Pipeline Services with HevoWe are seeking an experienced Data Engineering Manager to lead the development of scalable platform services supporting thousands of customers' data pipelines.This role offers a unique opportunity to build and own critical components of our cloud warehouse solution, ensuring reliable data loading and streamlining...


  • Bengaluru, Karnataka, India Data Inc Full time

    About the RoleWe are seeking a highly skilled Chief Data Engineering Lead to join our team at Data Inc.This is an exciting opportunity to lead the development and implementation of architecture strategy and standards pertinent to the domain, and establish Standards and Best practices.


  • Bengaluru, Karnataka, India HEVO DATA Full time

    Unlock Career Growth as a Data Engineering LeadWe're seeking an exceptional Data Engineering Lead to join our team at Hevo Data. In this role, you'll have the opportunity to own and build scalable platform services that support thousands of customers' data pipelines and ensure load data reliability to cloud warehouses.The Hevo platform core services will...

  • Data engineer

    1 month ago


    Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices.Deliver assigned projects within budgeted time, cost and agreed...


  • Bengaluru, Karnataka, India Data Labs - LANDMARK GROUP Full time

    About UsData Labs - LANDMARK GROUP is a leading data engineering firm committed to delivering cutting-edge analytics solutions.Job DescriptionWe are seeking an experienced Lead Data Engineering Manager to oversee the development and deployment of our analytics products. This role involves leading a team of engineers in designing, building, and deploying...

  • Data Engineer

    3 months ago


    Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and agreed...

  • Data Engineer

    3 months ago


    Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data Engineer Location: Yemalur, Bangalore Your Key Responsibilities: Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices. Deliver assigned projects within budgeted time, cost and agreed...

  • Data Engineer

    2 months ago


    Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.Lead the architecture, design, and delivery of enterprise grade microservices.Deliver assigned projects within budgeted time, cost and agreed...

  • Data Science

    4 weeks ago


    Bengaluru, Karnataka, India NTT DATA Full time

    Job OverviewWe are seeking an experienced Data Science & AI Engineer to lead our machine learning initiatives and drive innovation in artificial intelligence.About the RoleThe successful candidate will have a strong background in developing and implementing traditional machine learning algorithms, with experience deploying models in production environments....


  • Bengaluru, Karnataka, India HEVO DATA Full time

    Engineering Leadership OpportunityThis is a unique chance to own and build scalable platform services that support thousands of customers' data pipelines, ensuring load data reliability to cloud warehouses.The Hevo platform core services will provide no-code-low pipeline building capabilities, making a significant impact on the company's business.About the...


  • Bengaluru, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data EngineerLocation: Yemalur, BangaloreYour Key Responsibilities:Lead our engineering team successfully build and deploy cutting edge analytics products for consumption by different stakeholders.- Lead the architecture, design, and delivery of enterprise grade microservices.- Deliver assigned projects within budgeted time, cost and agreed...

  • Lead Data Architect

    3 weeks ago


    Bengaluru, Karnataka, India Data Labs - LANDMARK GROUP Full time

    Job Title: Lead Data ArchitectLocation: Yemalur, BangaloreAt Data Labs - LANDMARK GROUP, we are seeking a skilled Lead Data Architect to spearhead our engineering team in successfully designing and deploying cutting-edge analytics products.Main Responsibilities:Lead the architecture, design, and delivery of enterprise-grade microservices.Partner with Product...


  • Bengaluru, Karnataka, India Hevo Data Full time

    About Hevo Data:We are a leading provider of fully managed Automated Pipelines, used by thousands of data-driven companies worldwide. Our mission is to enable every company to be data-driven, and we're looking for an experienced Customer Experience Lead to join our team.As a Customer Experience Lead at Hevo Data, you will be responsible for ensuring...


  • Bengaluru, Karnataka, India NTT DATA Full time

    About the Role:-Sr Syniti ADM Consultant:As a highly experienced SAP Data Migration Consultant, you will lead the execution of data migration projects from SAP ECC to SAP S4/HANA, utilizing your expertise in data cleansing, profiling, and harmonization. Your proficiency in project planning, cutover planning, and gathering business requirements is essential...


  • Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: AI Data Science and Engineering LeaderAbout the Role:We are seeking a highly skilled Ai Data Science and Engineering Leader to join our team in Bangalore, India. This is an excellent opportunity for a talented individual to lead our data science and engineering efforts.Responsibilities:Develop and implement traditional machine learning algorithms...

  • GCP Lead Engineer

    4 weeks ago


    Bengaluru, Karnataka, India NTT DATA Full time

    About the RoleWe are seeking a skilled GCP Lead Engineer to join our team at NTT DATA. This role is ideal for someone with expertise in designing and implementing Google Cloud Platform (GCP) solutions.Key ResponsibilitiesAssess, design, and implement GCP solutions, including compute, network, storage, identity, security, and disaster recovery.Develop and...


  • Bengaluru, Karnataka, India NTT DATA Full time

    Job Title: Data Engineering Specialist - BangaloreAbout the Role:We are seeking a highly skilled Data Engineering Specialist to join our team in Bangalore, India. As a key member of our engineering team, you will be responsible for designing and developing high-quality data pipelines using ETL code in Databricks.Key Responsibilities:Develop high-level design...

  • AI Data Engineer

    1 month ago


    Bengaluru, India NTT DATA Full time

    Job Description Job Title: Data Science & AIML, GenAI Lead/EngineerKey Responsibilities:• Develop and implement traditional machine learning algorithms.• Deploy at least one model in a production environment.• Write and maintain Python code for data science and machine learning projects.Preferred Qualifications:• Knowledge of Deep Learning (DL)...


  • Bengaluru, Karnataka, India NTT DATA Full time

    About the RoleWe are seeking a skilled Data Engineer to join our team in Bangalore, India. As a key member of our data engineering team, you will play a crucial role in designing and implementing automated data pipelines.


  • Bengaluru, Karnataka, India NTT DATA Full time

    About the Role:As a Cloud Data Engineer, you will be responsible for designing and implementing large-scale data processing systems using GCP's BigQuery, Dataflow, and Dataproc technologies. Your expertise in ETL/ELT pipelines, data warehousing, and lakehouse solutions will enable us to efficiently process structured and unstructured data.Key...