Lead Data Engineer

3 days ago

pune, India Edgematics Full time

Job Description:


We are seeking an experienced Data Engineer to join our team and lead the development of a cutting-edge data platform. The platform will leverage distributed computing frameworks such as Apache Spark, Databricks, and Snowflake to enable near real time supply chain planning, eventually leading to advanced analytics, insights into data with the adoption of Generative AI (GenAI) technologies across our product base.


KEY RESPONSIBILITIES:

As a Lead Data Engineer, the candidate would be responsible for:

  • Design and build a highly scalable, fault-tolerant data platform optimized for distributed computing and large-scale data processing
  • Implement data pipelines and ETL/ELT processes using distributed computing frameworks to efficiently ingest, transform, and load massive datasets from various sources
  • Leverage cloud data platforms to enable seamless data sharing, near-zero maintenance, and fast analytics on structured and semi-structured data.
  • Collaborate with data scientists, machine learning engineers, and software developers to understand data requirements and build solutions to power GenAI applications
  • Optimize distributed computing jobs and queries for maximum performance and cost efficiency
  • Implement data governance, security, and compliance best practices
  • Provide guidance on distributed computing architecture and mentor junior data engineers


QUALIFICATIONS

  • 5+ years of experience as a Data Engineer working with big data technologies.
  • Strong proficiency in SQL, Python programming and data modeling techniques.
  • Deep expertise in distributed computing principles and frameworks (e.g., Apache Spark), including SQL, streaming, and optimizing jobs for scale and efficiency.
  • Hands-on experience with developing and deploying distributed computing applications using cloud-based platforms (e.g., AWS EMR, Azure HDInsight, or equivalent).
  • Strong understanding of cloud data platform architectures and best practices for ELT/ETL, data sharing, and query optimization (e.g., AWS Athena, AWS Glue, Azure Synapse Analytics, or equivalent).
  • Experience enabling application engineers to build applications leveraging the data platform through APIs and abstractions.
  • Experience with orchestration frameworks like Apache Airflow and data streaming technologies like Kafka
  • Experience building and optimizing data pipelines for machine learning applications.
  • Knowledge of data modelling, data warehousing, and schema design.
  • Familiarity with public cloud platforms such as AWS, Azure, or GCP.
  • Excellent problem-solving and communication skills.
  • Bachelor's or Master's degree in Computer Science (Preferred), Engineering, or a related field


EXPERIENCE: 5+ years


LOCATION: Pune

  • Technical Lead

    5 hours ago

    Pune, India Data Axle India Full time

    Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging...

  • Technical Lead

    1 month ago

    Pune, India Data Axle India Full time

    Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging...

  • Technical Lead

    1 month ago

    Pune, India Data Axle India Full time

    Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging...

  • Technical Lead

    1 month ago

    Pune, India Data Axle India Full time

    Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by leveraging...

  • Lead Engineer

    3 months ago

    Pune, India NTT DATA Full time

    Job Description Req ID: 286268 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Lead Engineer to join our team in Remote, Mahārāshtra (IN-MH), India (IN).Qualifications:...

  • Lead Data Scientist

    2 days ago

    Pune, India Data Axle India Full time

    About Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by...

  • Lead Data Scientist

    5 hours ago

    Pune, India Data Axle India Full time

    About Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales, and research for over 50 years in the USA. Data Axle now has an established strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and by...

  • Data Engineer

    1 week ago

    pune, India NTT DATA Services Full time

    Req ID: 297728 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Pune, Mahārāshtra (IN-MH), India (IN).Job Title Developer/ Data...

  • Data Engineer

    7 days ago

    pune, India NTT DATA Services Full time

    Req ID:  297728  NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer to join our team in Pune, Mahārāshtra (IN-MH), India (IN). Job Title  Developer/ Data Engineer...

  • Data Engineer

    5 hours ago

    Pune, India NTT DATA Services Full time

    Req ID:297728NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Pune, Mahārāshtra (IN-MH), India (IN).Job TitleDeveloper/ Data EngineerLocationPuneOur...

  • Digital Engineering Lead Engineer

    3 months ago

    Pune, India NTT DATA Full time

    Job Description Req ID: 286832 NTT DATA Services strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Digital Engineering Lead Engineer to join our team in Pune, Mahārāshtra (IN-MH), India...

  • Data Engineer

    1 week ago

    Pune, India NTT DATA Services Full time

    Req ID: 297728  NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer to join our team in Pune, Mahārāshtra (IN-MH), India (IN). Job Title Developer/ Data...

  • Data Engineer

    1 week ago

    Pune, India NTT DATA Services Full time

    Req ID: 297728 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Pune, Mahārāshtra (IN-MH), India (IN).Job Title Developer/ Data...

  • Senior Data Engineer

    4 months ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Senior Data Engineer

    4 months ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Senior Data Engineer

    1 week ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Digital Engineering Lead Engineer

    1 week ago

    Pune, India NTT DATA Services Full time

    Req ID: 296847  NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Digital Engineering Lead Engineer to join our team in Pune, Mahārāshtra (IN-MH), India (IN). Once You...

  • Lead Data Engineer

    4 days ago

    pune, India Edgematics Full time

    Job Description: We are seeking an experienced Data Engineer to join our team and lead the development of a cutting-edge data platform. The platform will leverage distributed computing frameworks such as Apache Spark, Databricks, and Snowflake to enable near real time supply chain planning, eventually leading to advanced analytics, insights into data with...

  • Lead Data Engineer

    4 months ago

    Pune, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that...

  • Lead Data Engineer

    3 days ago

    Pune, India Edgematics Full time

    Job Description:We are seeking an experienced Data Engineer to join our team and lead the development of a cutting-edge data platform. The platform will leverage distributed computing frameworks such as Apache Spark, Databricks, and Snowflake to enable near real time supply chain planning, eventually leading to advanced analytics, insights into data with the...