Lead Data Engineer/Manager

4 weeks ago

Bangalore, India Indium Software Full time

Job Description :


We are seeking a highly skilled Manager/Lead Data Engineer with extensive experience in building and maintaining robust data pipelines and ETL processes. The ideal candidate will have strong expertise in PySpark, Python, SQL, and workflow orchestration tools such as Airflow.


As a Manager/Lead Data Engineer, you will be responsible for leading data engineering efforts, collaborating with data analysts, and ensuring the delivery of high-quality, reliable data to support business insights and decision-making.


Key Responsibilities :

  • Technical Leadership: Lead a team of data engineers in designing, developing, and maintaining scalable and efficient data pipelines and ETL processes.
  • SQL Expertise: Demonstrate strong proficiency in SQL queries, including outer joins, aggregations, unions, window functions, and common table expressions (CTEs), to manipulate and analyze complex datasets efficiently.
  • Collaboration with Data Analysts: Work closely with data analysts to understand their requirements, provide necessary data, and guide them on data-related issues, ensuring alignment with business objectives.
  • ETL Development: Design and implement ETL processes using Hadoop, Hive, PySpark, Presto, SparkSQL, and other relevant technologies, ensuring data quality, consistency, and reliability.
  • Workflow Orchestration: Implement and manage workflow orchestration tools such as Airflow to schedule, monitor, and automate data pipelines and workflows, ensuring timely and accurate data delivery.
  • Stakeholder Management: Independently handle stakeholders and interact with them to understand their requirements, provide updates on data engineering initiatives, and address any concerns or issues, minimizing the need for reporting manager intervention.
  • Performance Optimization: Identify opportunities for performance optimization and efficiency improvements in data pipelines and ETL processes, implementing best practices and optimizations to enhance overall system performance.
  • Data Governance and Compliance: Ensure compliance with data governance policies, standards, and regulations, and implement necessary controls to safeguard data privacy and security.
  • Team Collaboration and Mentorship: Foster a collaborative team environment, providing guidance, mentorship, and support to junior team members, and promoting knowledge sharing and skill development.

Skills and Qualifications :

  • Bachelors/Masters degree in Computer Science, Engineering, or a related field.
  • Proven experience (6-10 years) in data engineering, with strong expertise in building and maintaining data pipelines and ETL processes.
  • Proficiency in programming languages such as Python, along with strong experience in PySpark for big data processing.
  • Strong knowledge and seasoned experience with SQL queries, including advanced concepts such as outer joins, aggregations, unions, window functions, and common table expressions (CTEs).
  • Experience working closely with data analysts, providing the necessary data and guidance on data-related issues.
  • Solid experience with ETL development using Hadoop, Hive, PySpark, Presto, SparkSQL, and related technologies.
  • Experience with workflow orchestration tools such as Airflow for scheduling and automating data pipelines and workflows.
  • Excellent communication and interpersonal skills, with the ability to interact effectively with stakeholders and collaborate with cross-functional teams.
  • Strong problem-solving skills and attention to detail, with a focus on delivering high-quality, reliable data solutions.
  • Leadership abilities, with a track record of successfully leading and managing data engineering projects and teams.
(ref:hirist.tech)

  • Data Engineering Lead

    4 weeks ago

    bangalore, India UNO Digital Bank Full time

    We are seeking a highly motivated and experienced Data Engineering Lead to join our young team. The ideal candidate will possess a deep understanding of data engineering principles, a proven track record of leading data engineering initiatives, the ability to drive the development and implementation of robust data infrastructure and an understanding of the...

  • Lead Data Engineer

    3 weeks ago

    bangalore, India Enverus Full time

    Description Enverus is a leading provider of energy analytics, trading & risk, and business automation solutions. We deliver business-critical insights to the global energy industry through a state-of-the-art SaaS platform built on industry-leading data and energy analytics. Our solutions deliver value across the entire energy value chain,...

  • Lead Data Engineer

    4 weeks ago

    bangalore, India Spectrum Consultants India Private Limited Full time

    Lead Data Engineer Summary Experience Required: 10 - 15 YearsJob Term: PermanentLocation: BangaloreCategory: Software DevelopmentKBS India serves as a hub for innovation supporting operations for Koch companies: Georgia-Pacific, Molex, Guardian, INVISTA, Koch Ag & Energy Solutions, Koch Minerals and Trading, Koch Business Solutions, KES, etc., Specialties...

  • Lead Data Engineering

    7 days ago

    bangalore, India Fractal Full time

    It's fun to work in a company where people truly BELIEVE in what they are doing! About Fractal : What makes Fractal a GREAT fit for you? When you join Fractal, you'll be part of a fast-growing team that helps our clients leverage AI together with the power of behavioral sciences to make better decisions. We're a strategic analytics partner to most admired...

  • Lead Data Engineer

    3 weeks ago

    bangalore, India Koch Global Services Full time

    Your Job The Lead Data Engineer will report to the Data Engineering & BI Lead of the KGSI and will be responsible to develop and implement a future-state data analytics platform for both the back-end data processing and the front-end data visualization component for the Finance Data Delivery teams. This will be a hands-on role and the candidate will be...

  • Data Engineering Lead

    4 weeks ago

    bangalore, India Mercedes-Benz Full time

    Tätigkeitsbereich:IT / TelekommunikationFachabteilung:HRITIC/DE HCM ERP ExtensionsGesellschaft:Mercedes-Benz Research and Development India Private LimitedStandort:Mercedes-Benz Research and Development India, BangaloreStartdatum:sofortVeröffentlichungsdatum:..4Stellennummer:MERX3Arbeitszeit:Vollzeit Join usAufgaben Description RESPONSIBILITITES -  ...

  • Senior Mechanical Engineer

    1 week ago

    bangalore, India Abha Engineer Full time

    We are looking for a Senior Mechanical EngineerRoles are described below.1. Manpower Planning.2. Preparing of Project Cost.3. Schedule wise work execution.4. As Drawing & quality work execution.5. Client & Third Party Manage.6. Working Team Manage & Review.7. Reporting to Management.8. ROB & FOB Fabrication & Erection Work Knowledge.

  • Lead Data Engineer

    4 weeks ago

    Bangalore, India Huron Full time

    The Opportunity Huron is redefining what a global consulting organization can be. Advancing new ideas every day to build even stronger clients, individuals and communities. We’re helping our clients find new ways to drive growth, enhance business performance and sustain leadership in the markets they serve. And, we’re developing strategies and...

  • Lead Data Engineer

    4 weeks ago

    bangalore, India Huron Full time

    The Opportunity Huron is redefining what a global consulting organization can be. Advancing new ideas every day to build even stronger clients, individuals and communities. We’re helping our clients find new ways to drive growth, enhance business performance and sustain leadership in the markets they serve. And, we’re developing strategies...

  • Data Engineering Lead

    2 weeks ago

    Bangalore, India hireveda Full time

    About Company : A series-c funded Ayurvedic health brand based in India. They offer a range of natural and Ayurvedic products.Role Description : As a Data Engineering Lead at our organization, you will be at the forefront of shaping and managing our data infrastructure with a primary focus on Google Cloud Platform (GCP). You will lead a team of data...

  • Lead Data Engineer

    1 week ago

    bangalore, India Vistaprint Full time

    Lead Data EngineerWho We Are:  Vista's Engineering organization is working to make our company one of the world's most well-known and successful data-driven companies. The cross-functional team includes analysts, software and data engineers, architects, data scientists, product owners, and more - all passionate about providing Vista with solutions,...

  • Lead Data Engineer

    4 weeks ago

    bangalore, India MakeMyTrip Full time

    Position: Lead Data Engineer / Principal Data Engineer Experience: Must have 6+ years of experience About Role: We are looking for experienced Data engineers with excellent problem-solving skills to develop machine-learning powered Data Products design to enhance customer experiences. About us: Nurtured from the seed of a single great idea - to empower the...

  • GCP Data Engineer

    6 days ago

    bangalore, India Infogain Full time

    GCP Data Engineer (Lead) with skills Big Data, Kafka, Python, Scala, Apache Spark for location Bangalore, India Posted on: May 08, Share on Linkedin Share on Twitter Share on Facebook ROLES & RESPONSIBILITIES Mandatory skills - Java8, Python, Spark, Spring Boot, GCP/Cloud, Jenkins, Terraform. Should have at least 8+ yrs of exp (for...

  • Lead Data Engineer

    3 weeks ago

    bangalore, India Cermati.com Full time

    Job DescriptionThe candidate will lead a technical team of 5-10 engineers in the data platform team . We can think of this role as that of a staff software engineer plus management .  Typical responsibilities will include People management, in the form of performance reviews, 1:1 session, mentoring and career growth for reportees Project management, in the...

  • Data Engineering Lead

    3 days ago

    bangalore, India In Time Tec Full time

    Exprience: 5+yearsJob Overview:The position requires an experienced and ambitious candidate who is passionate about technology and is self-driven. The candidate must be technically proficient in all phases of development life cycle, yet savvy enough to interface with non-technical stakeholders to achieve results and meet common project goals. The candidate...

  • Data Management Lead

    2 weeks ago

    bangalore, India 14260 GSK India Global Services Private Limited Full time

    Data Management Lead The PPM Digital Data Analytics and Performance (DDAP) team’s mission is to embed effective integrated R&D project management through data management standards, systems, and analytics; Increasing organizational digital capabilities; Enabling efficient project, resource, and pipeline management with good performance measures. ...

  • Tech Lead

    4 weeks ago

    bangalore, India StoneX Full time

    Overview Job Description Tech Lead – Data Reliability Engineering Reports to: Director Data Distribution Company: StoneX Group Inc. Position Purpose : As a Technical Lead in Data Reliability Engineering at StoneX you will guide our data platform and team of engineers to ensure the datasets are reliable, scalable and...

  • ADC Data Engineering Lead

    1 week ago

    bangalore, India Growel Softech Pvt. Ltd. Full time

    Job Description:We are seeking an experienced ADC (Advanced Data Computing) Data Engineering Lead with 10-13 years of experience to join our team. The ideal candidate will have a strong background in Big Data engineering, with expertise in tools such as Databricks, Azure Data Factory, and SQL. As an ADC Data Engineering Lead, you will be responsible for...

  • ADC Data Engineering Lead

    2 weeks ago

    Bangalore, India Growel Softech Pvt. Ltd. Full time

    Job Description:We are seeking an experienced ADC (Advanced Data Computing) Data Engineering Lead with 10-13 years of experience to join our team. The ideal candidate will have a strong background in Big Data engineering, with expertise in tools such as Databricks, Azure Data Factory, and SQL. As an ADC Data Engineering Lead, you will be responsible for...

  • ADC Data Engineering Lead

    2 weeks ago

    Bangalore, Karnataka, India Growel Softech Pvt. Ltd. Full time

    Job Description:We are seeking an experienced ADC (Advanced Data Computing) Data Engineering Lead with 10-13 years of experience to join our team. The ideal candidate will have a strong background in Big Data engineering, with expertise in tools such as Databricks, Azure Data Factory, and SQL. As an ADC Data Engineering Lead, you will be responsible for...