Lead Big Data Engineer

3 months ago

Gurugram, India Huntingcube Recruitment Solutions Full time

Job Description :


- Lead the design and implementation of scalable and efficient big data architectures, ensuring optimal performance, security, and scalability.


- Design, build, and maintain complex data pipelines that process large volumes of data from multiple sources, ensuring data is clean, reliable, and accessible.

- Implement and manage big data storage solutions using technologies like Hadoop, Apache Spark, and cloud-based data lakes (e., AWS S3, Azure Data Lake).

- Develop and optimize Extract, Transform, Load (ETL) processes to ensure efficient data movement and transformation across the data pipeline.

- Implement data quality checks and monitoring mechanisms to ensure data integrity, consistency, and accuracy.

- Optimize data processing jobs for performance and cost efficiency, identifying and resolving bottlenecks in data processing workflows.

- Integrate big data solutions with existing data warehouses, databases, and other enterprise systems, ensuring seamless data flow across the organization.

- Lead and mentor a team of data engineers, providing guidance on best practices, coding standards, and architectural design.

- Work closely with data scientists, data analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

- Document data engineering processes, architecture designs, and best practices to ensure knowledge sharing and consistency across the team.

Innovation and Continuous Improvement :


- Stay updated with the latest trends and advancements in big data technologies, and continuously improve existing data solutions and processes.

Technical Requirements :

- Strong expertise in big data technologies such as Hadoop, Apache Spark, Kafka, HBase, and Hive.

- Experience with data processing frameworks like Apache Flink, Apache Beam, or similar technologies.

- Proficiency in cloud platforms such as AWS, Azure, or Google Cloud, with hands-on experience in cloud-based big data services (e., EMR, Databricks, BigQuery).

- Strong programming skills in languages such as Python, Java, or Scala, with a focus on data engineering and big data processing.

- Experience with data warehousing solutions like Amazon Redshift, Google BigQuery, or Snowflake.

- Strong knowledge of NoSQL databases (e., Cassandra, MongoDB) and relational databases (e., MySQL, PostgreSQL).

- Expertise in data modeling, schema design, and data partitioning strategies for large-scale data systems.

- Experience with ETL tools and frameworks such as Apache NiFi, Talend, or Informatica.

- Knowledge of data security best practices, including data encryption, access control, and compliance with data protection regulations.

- Proficiency with version control systems like Git and familiarity with CI/CD pipelines for data engineering workflows.

Qualifications :

- Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, or a related field.

- 7 to 10 years of hands-on experience in big data engineering, with a proven track record of delivering high-quality, scalable data solutions

(ref:hirist.tech)

  • Lead Data Engineer – Big Data

    2 months ago

    gurugram, India Incedo Inc. Full time

    Role Description: Sr. Lead Data Engineer – Big Data The ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated person with a passion for problem solving and continuous learning....

  • Lead Data Engineer – Big Data

    2 months ago

    Gurugram, India Incedo Inc. Full time

    Role Description: Sr. Lead Data Engineer – Big DataThe ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated person with a passion for problem solving and continuous...

  • Lead Data Engineer – Big Data

    2 months ago

    Gurugram, India Incedo Inc. Full time

    Role Description: Sr. Lead Data Engineer – Big DataThe ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated person with a passion for problem solving and continuous...

  • Lead Data Engineer – Big Data

    2 months ago

    gurugram, India Incedo Inc. Full time

    Role Description: Sr. Lead Data Engineer – Big DataThe ideal candidate is a hands-on technology developer with experience in developing scalable applications and platforms. They must be at ease working in an agile environment with little supervision. The person should be a self-motivated person with a passion for problem solving and continuous...

  • Lead Big Data Engineer

    1 month ago

    Gurgaon/Gurugram, IN Huntingcube Recruitment Solutions Full time

    Job Description : - Lead the design and implementation of scalable and efficient big data architectures, ensuring optimal performance, security, and scalability.- Design, build, and maintain complex data pipelines that process large volumes of data from multiple sources, ensuring data is clean, reliable, and accessible.- Implement and manage big data storage...

  • Lead Big Data Engineer

    4 weeks ago

    Gurgaon/Gurugram, India Huntingcube Recruitment Solutions Full time

    Job Description : - Lead the design and implementation of scalable and efficient big data architectures, ensuring optimal performance, security, and scalability. - Design, build, and maintain complex data pipelines that process large volumes of data from multiple sources, ensuring data is clean, reliable, and accessible.- Implement and manage big data...

  • Big Data Engineer

    1 week ago

    Gurgaon/Gurugram/Bangalore/Hyderabad, India Huquo Full time

    Huquo is seeking a Big Data Engineer to join our team.We are looking for a skilled professional with expertise in Hadoop ecosystem (HDFS, Hive, Yarn, File formats like Avro/Parquet) and Python programming language. Additionally, you should have experience with PySpark, SQL, and Agile Methodology.Key Responsibilities:Develop and maintain large-scale data...

  • Big Data Engineer

    3 weeks ago

    Pune/Gurgaon/Gurugram, India Huquo Full time

    Job Title: Big Data EngineerAt Huquo, we are seeking a highly skilled Big Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data processing systems.Key Responsibilities:Design and develop scalable data pipelines using Spark and other big data...

  • Big Data Engineer

    3 weeks ago

    Chennai/Noida/Gurgaon/Gurugram/Mumbai, India ANRI Solutions HR Services Full time

    Job Title: Big Data EngineerWe are seeking an experienced Big Data Engineer to join our team at ANRI Solutions HR Services.Key Responsibilities:Design and develop end-to-end data pipelines using Spark and Scala.Develop expertise in ETL and data warehousing concepts, including cloud integration.Work with minimal guidance and lead the team...

  • Lead data engineer

    2 months ago

    gurugram, India Incedo Inc. Full time

    Role Description:As a Senior Technical Lead - Big Data Platform at Incedo, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects. You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure. You will be skilled in big data...

  • Big Data Engineer

    1 month ago

    Gurgaon/Gurugram, India Peoplestrong HR Full time

    Job Title : Big Data Engineer (SQL Server) Must have:- Power BI, SISS or SSRSLocation : Gurgaon (Work From Office)Qualification and Requirements : Educational Background : Bachelor's degree in Computer Science, Information Technology, or a related field.Experience : 2-4 years of experience as a Database Developer, Database Engineer, or in a similar...

  • Lead data engineer

    2 months ago

    Gurugram, India Incedo Inc. Full time

    Role Description: As a Senior Technical Lead - Big Data Platform at Incedo, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects. You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure. You will be skilled in big...

  • Lead data engineer

    2 months ago

    Gurugram, India Incedo Inc. Full time

    Role Description:As a Senior Technical Lead - Big Data Platform at Incedo, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects. You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure. You will be skilled in big data...

  • Lead data engineer

    2 months ago

    Gurugram, India Incedo Inc. Full time

    Role Description:As a Senior Technical Lead - Big Data Platform at Incedo, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects. You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure. You will be skilled in big data...

  • Lead data engineer

    2 months ago

    gurugram, India Incedo Inc. Full time

    Role Description: As a Senior Technical Lead - Big Data Platform at Incedo, you will be responsible for designing and implementing big data platforms to support large-scale data integration projects. You will work with data architects and data engineers to define the platform architecture and build the necessary infrastructure. You will be skilled in big...

  • Big Data Engineer

    3 weeks ago

    Gurgaon/Gurugram/Bangalore/Hyderabad, India Huquo Full time

    Job Title: Big Data EngineerJob Summary:Huquo is seeking a highly skilled Big Data Engineer to join our team. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Hadoop ecosystem tools such as HDFS, Hive, Yarn, and File formats like Avro/Parquet.Key Responsibilities:Design and...

  • Manager Data Engineering DE

    4 weeks ago

    Gurugram, India Publicis Sapient Full time

    Job DescriptionAs Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing AWS Cloud-based big data solutions for clients. Your role will be focused on delivering high-quality solutions by independently driving design discussions related to Data Ingestion, Transformation & Consumption,...

  • Big Data Engineer

    3 weeks ago

    Chennai/Noida/Gurgaon/Gurugram/Mumbai, India ANRI Solutions HR Services Full time

    Job Title: Big Data EngineerWe are seeking an experienced Big Data Engineer to join our team at ANRI Solutions HR Services. The ideal candidate will have expertise in Spark and Scala, as well as a strong understanding of data warehousing and cloud concepts.Key Responsibilities:Design and develop end-to-end data pipelines using Spark and ScalaDevelop and...

  • Big Data Developer

    4 weeks ago

    Gurugram, India Shya Workforce Solutions Full time

    • Collaborate with enterprise architects, data architects, developers & engineers, datascientists, and information designers to lead the identification and definition of requireddata structures, formats, pipelines, metadata, and workload orchestration capabilities• Experience in service architecture, development, and high performance and scalability.•...

  • Big Data Developer

    4 weeks ago

    gurugram, India Shya Workforce Solutions Full time

    • Collaborate with enterprise architects, data architects, developers & engineers, data scientists, and information designers to lead the identification and definition of required data structures, formats, pipelines, metadata, and workload orchestration capabilities • Experience in service architecture, development, and high performance and scalability....