Big Data Engineer

4 weeks ago

Gurgaon, Haryana, India One Click AI Full time

Job Title : Big Data Engineer (Hadoop/Spark)

Experience : 36 Years

Location : Noida / Gurugram (Hybrid or On-site, as applicable)

Notice Period : 30 Days

Employment Type : Summary :

We are looking for a skilled and passionate Big Data Engineer with strong expertise in the Hadoop ecosystem and Apache Spark to join our growing data engineering team. The ideal candidate will design, develop, and maintain scalable data pipelines and architectures that power advanced analytics and data-driven Responsibilities :- Design and implement distributed data processing systems using Hadoop, Spark, and related technologies.

- Build robust ETL pipelines for structured and unstructured data ingestion, transformation, and loading.

- Optimize and tune Spark jobs for performance and scalability across large datasets.

- Work with data scientists, analysts, and stakeholders to understand data requirements and deliver efficient solutions.

- Integrate data from various sources such as RDBMS, NoSQL, APIs, flat files, or cloud storage.

- Ensure data quality, integrity, and security through validation, monitoring, and alerting systems.

- Manage and maintain HDFS, Hive, Spark clusters, and job workflows using Oozie, Airflow, or similar schedulers.

- Support real-time/streaming data processing using Kafka, Spark Streaming, or Flink (as applicable).

- Collaborate in Agile development cycles and participate in code reviews, sprint planning, and retrospectives.

- Document data flows, architecture decisions, and operational Skills & Qualifications :

- 36 years of hands-on experience in Big Data engineering roles.

- Strong programming skills in Python, Scala, or Java.

- Proficiency with Hadoop ecosystem tools: HDFS, Hive, Pig, HBase, Sqoop, etc.

- Advanced working knowledge of Apache Spark for both batch and real-time processing.

- Solid understanding of data modeling, partitioning, bucketing, and optimization techniques.

- Experience with workflow orchestration tools like Apache Airflow, Oozie, or Luigi.

- Familiarity with cloud platforms like AWS, GCP, or Azure and services like EMR, Databricks, or BigQuery is a plus.

- Hands-on experience with version control (e.g., Git), CI/CD, and containerization (Docker/Kubernetes) is desirable.

- Excellent problem-solving and communication Qualifications :

- Bachelors or Masters degree in Computer Science, Engineering, or a related field.

- Certifications in Big Data (e.g., Cloudera, Databricks, AWS Big Data Specialty) are a plus.

- Experience working in data lake or lakehouse architectures is a Join Us ?

- Work with a dynamic team on cutting-edge big data platforms and cloud technologies.

- Competitive compensation and performance-based incentives.

- Flexible and collaborative work environment.

- Opportunities for upskilling and professional growth.

(ref:hirist.tech)

  • Big Data Modernization Engineer

    15 hours ago

    Gurgaon, Haryana, India Infosys Limited Full time

    Job Description- Engineer has to help Director in Modernization- Convert Talend jobs into PySpark or Data proc jobs- Big data skills- Good in written and verbal Communication- Good problem-solving skills and programming skills

  • Senior Big Data Infrastructure Modernizer

    6 hours ago

    Gurgaon, Haryana, India beBeeDataTransformation Full time ₹ 15,00,000 - ₹ 20,00,000

    Big Data Transformation SpecialistModernize our data infrastructure and drive business innovation by leveraging advanced big data skills.Key Responsibilities:Collaborate with senior leaders to develop a comprehensive strategy for modernizing big data systems.Translate Talend jobs into PySpark or Data proc jobs to enhance efficiency and scalability.Apply...

  • Big Data Engineering Manager| Ml Ops

    3 weeks ago

    Gurgaon, Haryana, India DUNNHUMBY Full time

    dunnhumby is the global leader in Customer Data Science empowering businesses everywhere to compete and thrive in the modern data-driven economy We always put the Customer First Our mission to enable businesses to grow and reimagine themselves by becoming advocates and champions for their Customers With deep heritage and expertise in retail - one of the...

  • Data Engineer

    3 weeks ago

    Gurgaon, Haryana, India Staffice Global Private Limited Full time

    Job DescriptionThis role is for a Data Engineer with extensive experience in data engineering, data warehousing, and big data processing. The ideal candidate will have strong expertise in Python, PySpark, and AWS data services to design, develop, and maintain robust data pipelines.Key Responsibilities- Design and implement end-to-end data engineering...

  • Engineer - Data Management / Big Data

    2 days ago

    Gurgaon, Haryana, India American Express Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    At American Express, our culture is built on a 175-year history of innovation, shared values and Leadership Behaviors, and an unwavering commitment to back our customers, communities, and colleagues. As part of Team Amex, you'll experience this powerful backing with comprehensive support for your holistic well-being and many opportunities to learn new...

  • Data Engineer

    4 weeks ago

    Gurgaon, Haryana, India ORBION INFOTECH Full time

    Job Description :Key Responsibilities :- Data Architecture & Strategy : Design and build scalable and reliable data architectures, data warehouses, and data lakes.- ETL/ELT Pipeline Development : Architect, build, and optimize complex ETL/ELT processes to ingest, transform, and load data from diverse sources.- Cloud Platform Expertise : Leverage extensive...

  • Gcp Data Engineer

    10 hours ago

    Gurgaon, Haryana, India Infosys Limited Full time

    Job Description- Experienced Data Engineer with hands on experience on GCP offerings experienced in BigQuery/BigTable/Pyspark- Worked on prior data engineering projects leveraging GCP product offerings- Strong SQL background- Prior Amex experience is a big Plus

  • Data Engineer

    4 weeks ago

    Gurgaon, Haryana, India Policybazaar Full time

    Job DescriptionAs a data engineer -YOU GET TO WORK ON- Data modeling for OLAP, warehousing and building ETL pipelines on cloud- AWS technologies like EMR, Lambda, S3, SQS, EC2, Glue, etc- Using Airflow for orchestrationBASIC QUALIFICATIONS- 1-3 years of data engineering experience- Experience with SQL and NoSQL databases- Practical Knowledge of Linux or Unix...

  • Data Pipeline Engineer

    21 hours ago

    Gurgaon, Haryana, India beBeeDataPipeline Full time ₹ 9,00,000 - ₹ 12,00,000

    Job SummaryWe are seeking a highly skilled Data Pipeline Engineer to join our team. In this role, you will be responsible for designing, building, and maintaining complex data pipelines that enable us to extract insights from large datasets.About the RoleThis is a fantastic opportunity for someone who has experience with ETL tools and technologies, as well...

  • Senior Data Engineer

    21 hours ago

    Gurgaon, Haryana, India beBeeData Full time US$ 90,000 - US$ 1,20,000

    Job Description:As a senior data engineer, you will play a key role in designing and implementing scalable data architectures to meet the organization's business needs. You will be responsible for developing and maintaining large-scale data systems, ensuring high-performance, reliability, and security.The ideal candidate will have strong experience with big...