Big Data Engineer

6 days ago


gurugram, India Intraedge Technologies Ltd. Full time

Job Description :

We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.

Key Responsibilities :

GCP Data Engineering :

- Design, implement, and maintain data processing pipelines using GCP services such as BigQuery, DataProc, Airflow, Cloud Composer, and GCP Hydra Services.

- Optimize and manage data storage and retrieval processes to ensure efficient performance and cost-effectiveness.

Big Data Technologies :

- Utilize Hive, PySpark, and Python to develop and manage Big Data solutions.

- Implement data transformations, aggregations, and data enrichment processes.

Data Pipeline Development :

- Develop and maintain ETL processes to integrate and process large datasets from various sources.

- Ensure data quality, reliability, and security in all data processing activities.

Workflow Automation :

- Use Airflow and Cloud Composer to automate data workflows and orchestrate complex data processing tasks.

- Monitor and troubleshoot data pipelines to ensure smooth and uninterrupted operations.

Performance Optimization:

- Optimize data processing jobs for performance and scalability.

- Implement best practices for data storage, indexing, and partitioning in BigQuery and other GCP services.

Collaboration and Support :

- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.

- Provide technical support and troubleshooting for data-related issues.

Documentation and Best Practices :

- Maintain comprehensive documentation of data pipelines, workflows, and processes.

- Adhere to industry best practices and standards in Big Data and cloud engineering.

Required Skills and Qualifications :

- Education: Bachelor's degree in Computer Science, Information Technology, or a related field.

Experience :

- 3+ years of hands-on experience with GCP services such as BigQuery, DataProc, Airflow, Cloud Composer, and GCP Hydra Services.

- 3-8 years of experience in Big Data engineering.

Technical Skills :

- Proficient in Hive, PySpark, and Python.

- Strong understanding of data warehousing concepts and best practices.

- Experience with cloud data optimization and cost management.

Soft Skills :

- Strong problem-solving and analytical skills.

- Excellent communication skills, both verbal and written.

- Ability to work effectively in a team-oriented, collaborative environment.

Personal Attributes :

- Self-motivated, proactive, and able to work independently.

- Strong attention to detail and commitment to delivering high-quality solutions.

(ref:hirist.tech)
  • Big Data Engineer

    6 days ago


    Gurugram, India Virtusa Full time

    Big Data Engineer - CREQ191392 Description Primary Skills SQL, Big data, Hive, Hadoop (Advanced level) Secondary Skills Spark, Unix Shell scripting , Data warehousing concepts Experience in debugging and optimized the SQLs and ETL scripts Prior experience with Amex would be an advantage. Preferred Experience level : 3-5 yrs. Experience with CI/CD process...

  • Big Data Engineer

    6 days ago


    gurugram, India Virtusa Full time

    Big Data Engineer - CREQ191392 Description Primary Skills SQL, Big data, Hive, Hadoop (Advanced level)Secondary Skills Spark, Unix Shell scripting , Data warehousing conceptsExperience in debugging and optimized the SQLs and ETL scriptsPrior experience with Amex would be an advantage.Preferred Experience level : 3-5 yrs.Experience with CI/CD process and...

  • Data Engineer

    1 day ago


    Gurugram, India Mantra Labs Pvt Ltd Full time

    About the Role :We are seeking a talented Data Engineer to join our team and play a crucial role in developing and maintaining our data pipelines. You will be responsible for designing, implementing, and optimizing data processing pipelines using PySpark and other big data technologies. You will also be responsible for ensuring data quality and integrity...


  • gurugram, India PAR Technology Full time

    Title Senior Big Data Engineer Reporting Manager, Big Data Job location Jaipur/Gurugram About the role Sr Big Data Engineer is critical in leading Punchh’s data innovations. By leveraging prior industrial experience in big data, he/she will help create cutting-edge data and analytics products for Punchh’s business partners. This role requires close...


  • Gurugram, India PAR Technology Full time

    TitleSenior Big Data EngineerReportingManager, Big DataJob locationJaipur/GurugramAbout the role Sr Big Data Engineer is critical in leading Punchh’s data innovations. By leveraging prior industrial experience in big data, he/she will help create cutting-edge data and analytics products for Punchh’s business partners.This role requires close...


  • Gurugram, India PAR Technology Full time

    TitleSenior Big Data EngineerReportingManager, Big DataJob locationJaipur/GurugramAbout the role Sr Big Data Engineer is critical in leading Punchh’s data innovations. By leveraging prior industrial experience in big data, he/she will help create cutting-edge data and analytics products for Punchh’s business partners.This role requires close...


  • Gurugram, India Partech Full time

    Hi there! We’re PAR and our purpose is:To deliver solutions that connect people to the restaurants, meals and moments they love.We take that responsibility very seriously. As a leading provider of technology to the top restaurant brands in the world we’re calling all rebels, instigators, idealists and builders to join our constantly growing team!PAR...

  • Data Engineer

    18 hours ago


    Gurgaon/Gurugram, India Mantra Labs Pvt Ltd Full time

    About the Role :We are seeking a talented Data Engineer to join our team and play a crucial role in developing and maintaining our data pipelines. You will be responsible for designing, implementing, and optimizing data processing pipelines using PySpark and other big data technologies. You will also be responsible for ensuring data quality and integrity...

  • Data Engineer

    2 days ago


    Gurgaon/Gurugram, IN Mantra Labs Pvt Ltd Full time

    About the Role :We are seeking a talented Data Engineer to join our team and play a crucial role in developing and maintaining our data pipelines. You will be responsible for designing, implementing, and optimizing data processing pipelines using PySpark and other big data technologies. You will also be responsible for ensuring data quality and integrity...

  • Big Data Engineer

    2 weeks ago


    Bangalore/Gurgaon/Gurugram, India HR BIZ HUB Full time

    Job Title : Big Data DeveloperJob Description : Bachelor's degree in Engineering or Computer Science or equivalent OR Master's in Computer Applications or equivalent. Solid Experience of software development experience and leading teams of engineers and scrum teams. 4+ years of handson experience of working with Map-Reduce, Hive, Spark (core, SQL and...

  • Big Data Engineer

    4 weeks ago


    gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...

  • Big Data Engineer

    4 weeks ago


    Gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...

  • Big Data Engineer

    2 months ago


    Gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...


  • gurugram, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...


  • gurugram, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...

  • Publicis Sapient

    4 weeks ago


    Gurugram, India Publicis Sapient Full time

    Job Description :As Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing Azure Cloud based big data solutions for clients. Your role will be focused on delivering high quality solutions by independently driving design discussions related to Data Ingestion, Transformation &...

  • Big Data Engineer

    1 week ago


    gurugram, India Virtusa Full time

    Big Data Engineer - CREQ190592 Description Qualifications Bachelors degree in Engineering or Computer Science or equivalent OR Masters in Computer Applications or equivalent. 6+ years of software development experience and leading teams of engineers and scrum teams 3+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and...

  • Big Data Engineer

    1 month ago


    Gurugram, India Intraedge Technologies Ltd. Full time

    Job Description :We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.Key...

  • Big Data Engineer

    4 weeks ago


    Gurugram, India Intraedge Technologies Ltd. Full time

    Job Description :We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.Key...


  • Gurgaon/Gurugram, India Paisabazaar.com Full time

    Job Description : Qualifications for Data Engineer : 3+ Years of experience in building and optimizing 'big data' solutions required to fulfill business and technology requirements. 3+ years of technical expertise in areas of design and implementation using big data technology Hadoop, Hive, Spark, Python/Java. Strong analytic skills to understand...