Big Data Engineer

1 month ago

Gurugram, India Intraedge Technologies Ltd. Full time

Job Description :

We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.

Key Responsibilities :

GCP Data Engineering :

- Design, implement, and maintain data processing pipelines using GCP services such as BigQuery, DataProc, Airflow, Cloud Composer, and GCP Hydra Services.

- Optimize and manage data storage and retrieval processes to ensure efficient performance and cost-effectiveness.

Big Data Technologies :

- Utilize Hive, PySpark, and Python to develop and manage Big Data solutions.

- Implement data transformations, aggregations, and data enrichment processes.

Data Pipeline Development :

- Develop and maintain ETL processes to integrate and process large datasets from various sources.

- Ensure data quality, reliability, and security in all data processing activities.

Workflow Automation :

- Use Airflow and Cloud Composer to automate data workflows and orchestrate complex data processing tasks.

- Monitor and troubleshoot data pipelines to ensure smooth and uninterrupted operations.

Performance Optimization:

- Optimize data processing jobs for performance and scalability.

- Implement best practices for data storage, indexing, and partitioning in BigQuery and other GCP services.

Collaboration and Support :

- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.

- Provide technical support and troubleshooting for data-related issues.

Documentation and Best Practices :

- Maintain comprehensive documentation of data pipelines, workflows, and processes.

- Adhere to industry best practices and standards in Big Data and cloud engineering.

Required Skills and Qualifications :

- Education: Bachelor's degree in Computer Science, Information Technology, or a related field.

Experience :

- 3+ years of hands-on experience with GCP services such as BigQuery, DataProc, Airflow, Cloud Composer, and GCP Hydra Services.

- 3-8 years of experience in Big Data engineering.

Technical Skills :

- Proficient in Hive, PySpark, and Python.

- Strong understanding of data warehousing concepts and best practices.

- Experience with cloud data optimization and cost management.

Soft Skills :

- Strong problem-solving and analytical skills.

- Excellent communication skills, both verbal and written.

- Ability to work effectively in a team-oriented, collaborative environment.

Personal Attributes :

- Self-motivated, proactive, and able to work independently.

- Strong attention to detail and commitment to delivering high-quality solutions.

(ref:hirist.tech)

  • Big Data Engineer

    4 weeks ago

    gurugram, India OmegaHires Full time

    Job Title: Big Data Engineer  Duration: Full-time on OmegaHires Payroll  Location: Gurugram (Should be located near NCR to attend office for 3 days a week)   Should understand the  Credit Card domain and good to have knowledge on Compliance & Operational Risk . Good analytics experience to do data discovery, analyze the gaps in data, partner with Data...

  • Big Data Engineer

    5 days ago

    Bangalore/Gurgaon/Gurugram, India HR BIZ HUB Full time

    Job Title : Big Data DeveloperJob Description : Bachelor's degree in Engineering or Computer Science or equivalent OR Master's in Computer Applications or equivalent. Solid Experience of software development experience and leading teams of engineers and scrum teams. 4+ years of handson experience of working with Map-Reduce, Hive, Spark (core, SQL and...

  • Big Data Engineer

    1 month ago

    Gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...

  • Big Data Engineer

    3 weeks ago

    gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...

  • Big Data Engineer

    3 weeks ago

    Gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...

  • Big Data Engineer

    1 month ago

    gurugram, India dunnhumby Full time

    We’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...

  • Lead Big Data Software Engineer

    2 months ago

    gurugram, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...

  • Senior Big Data Software Engineer

    2 months ago

    gurugram, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...

  • Publicis Sapient

    3 weeks ago

    Gurugram, India Publicis Sapient Full time

    Job Description :As Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing Azure Cloud based big data solutions for clients. Your role will be focused on delivering high quality solutions by independently driving design discussions related to Data Ingestion, Transformation &...

  • Big Data Engineer

    1 day ago

    gurugram, India Virtusa Full time

    Big Data Engineer - CREQ190592 Description Qualifications Bachelors degree in Engineering or Computer Science or equivalent OR Masters in Computer Applications or equivalent. 6+ years of software development experience and leading teams of engineers and scrum teams 3+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and...

  • Big Data Engineer

    1 month ago

    gurugram, India Intraedge Technologies Ltd. Full time

    Job Description :We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.Key...

  • Big Data Engineer

    3 weeks ago

    Gurugram, India Intraedge Technologies Ltd. Full time

    Job Description :We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.Key...

  • Paisabazaar - Data Engineer - Big Data Technologies

    3 weeks ago

    Gurgaon/Gurugram, India Paisabazaar.com Full time

    Job Description : Qualifications for Data Engineer : 3+ Years of experience in building and optimizing 'big data' solutions required to fulfill business and technology requirements. 3+ years of technical expertise in areas of design and implementation using big data technology Hadoop, Hive, Spark, Python/Java. Strong analytic skills to understand...

  • Paisabazaar - Data Engineer - Big Data Technologies

    2 months ago

    Gurgaon/Gurugram, India Paisabazaar.com Full time

    Job Description : Qualifications for Data Engineer : 3+ Years of experience in building and optimizing 'big data' solutions required to fulfill business and technology requirements. 3+ years of technical expertise in areas of design and implementation using big data technology Hadoop, Hive, Spark, Python/Java. Strong analytic skills to understand...

  • Publicis Sapient

    4 weeks ago

    Gurugram, India Publicis Sapient Full time

    Job Description :As Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing Azure Cloud based big data solutions for clients. Your role will be focused on delivering high quality solutions by independently driving design discussions related to Data Ingestion, Transformation &...

  • Big Data Developer

    3 weeks ago

    Gurugram, India Shya Workforce Solutions Full time

    Collaborate with enterprise architects, data architects, developers & engineers, data scientists, andinformation designers to lead the identification and definition of required data structures, formats,pipelines, metadata, and workload orchestration capabilitiesExperience in service architecture, development, and high performance and scalability.Experience...

  • Big Data Developer

    3 weeks ago

    Gurugram, India Shya Workforce Solutions Full time

    Collaborate with enterprise architects, data architects, developers & engineers, data scientists, andinformation designers to lead the identification and definition of required data structures, formats,pipelines, metadata, and workload orchestration capabilitiesExperience in service architecture, development, and high performance and scalability.Experience...

  • Big Data Developer

    3 weeks ago

    gurugram, India Shya Workforce Solutions Full time

    Collaborate with enterprise architects, data architects, developers & engineers, data scientists, and information designers to lead the identification and definition of required data structures, formats, pipelines, metadata, and workload orchestration capabilities Experience in service architecture, development, and high performance and scalability....

  • Big Data Developer

    2 months ago

    Gurugram, India Rocket Consulting Full time

    Big Data EngineerPosition Overview :We are seeking a skilled Big Data Engineer to join our dynamic team. The ideal candidate will have expertise in designing, implementing, and maintaining large-scale data processing systems using cutting-edge technologies. As a Big Data Engineer, you will play a key role in building and optimizing data pipelines, data...

  • Big Data Developer

    2 months ago

    gurugram, India Rocket Consulting Full time

    Big Data EngineerPosition Overview :We are seeking a skilled Big Data Engineer to join our dynamic team. The ideal candidate will have expertise in designing, implementing, and maintaining large-scale data processing systems using cutting-edge technologies. As a Big Data Engineer, you will play a key role in building and optimizing data pipelines, data...