![Intraedge Technologies Ltd.](https://media.trabajo.org/img/noimg.jpg)
Big Data Engineer
3 weeks ago
Job Description :
We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.
Key Responsibilities :
GCP Data Engineering :
- Design, implement, and maintain data processing pipelines using GCP services such as BigQuery, DataProc, Airflow, Cloud Composer, and GCP Hydra Services.
- Optimize and manage data storage and retrieval processes to ensure efficient performance and cost-effectiveness.
Big Data Technologies :
- Utilize Hive, PySpark, and Python to develop and manage Big Data solutions.
- Implement data transformations, aggregations, and data enrichment processes.
Data Pipeline Development :
- Develop and maintain ETL processes to integrate and process large datasets from various sources.
- Ensure data quality, reliability, and security in all data processing activities.
Workflow Automation :
- Use Airflow and Cloud Composer to automate data workflows and orchestrate complex data processing tasks.
- Monitor and troubleshoot data pipelines to ensure smooth and uninterrupted operations.
Performance Optimization:
- Optimize data processing jobs for performance and scalability.
- Implement best practices for data storage, indexing, and partitioning in BigQuery and other GCP services.
Collaboration and Support :
- Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions.
- Provide technical support and troubleshooting for data-related issues.
Documentation and Best Practices :
- Maintain comprehensive documentation of data pipelines, workflows, and processes.
- Adhere to industry best practices and standards in Big Data and cloud engineering.
Required Skills and Qualifications :
- Education: Bachelor's degree in Computer Science, Information Technology, or a related field.
Experience :
- 3+ years of hands-on experience with GCP services such as BigQuery, DataProc, Airflow, Cloud Composer, and GCP Hydra Services.
- 3-8 years of experience in Big Data engineering.
Technical Skills :
- Proficient in Hive, PySpark, and Python.
- Strong understanding of data warehousing concepts and best practices.
- Experience with cloud data optimization and cost management.
Soft Skills :
- Strong problem-solving and analytical skills.
- Excellent communication skills, both verbal and written.
- Ability to work effectively in a team-oriented, collaborative environment.
Personal Attributes :
- Self-motivated, proactive, and able to work independently.
- Strong attention to detail and commitment to delivering high-quality solutions.
-
Big Data Engineer
2 weeks ago
gurugram, India OmegaHires Full timeJob Title: Big Data Engineer Duration: Full-time on OmegaHires Payroll Location: Gurugram (Should be located near NCR to attend office for 3 days a week) Should understand the Credit Card domain and good to have knowledge on Compliance & Operational Risk . Good analytics experience to do data discovery, analyze the gaps in data, partner with Data...
-
Big Data Engineer
4 weeks ago
gurugram, India dunnhumby Full timeWe’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...
-
Big Data Engineer
1 month ago
Gurugram, India dunnhumby Full timeWe’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...
-
Big Data Engineer
1 week ago
gurugram, India dunnhumby Full timeWe’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...
-
Big Data Engineer
1 week ago
Gurugram, India dunnhumby Full timeWe’re looking for a Big Data Engineer who expects more from their career. It’s a chance to extend and improve dunnhumby’s Data Engineering Team It’s an opportunity to work with a market-leading business to explore new opportunities for us and influence global retailers. What we expect from you • 2+ years of experience in Information...
-
Lead Big Data Software Engineer
2 months ago
gurugram, India Epam Full timeDescription EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...
-
Senior Big Data Software Engineer
2 months ago
gurugram, India Epam Full timeDescription EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...
-
Publicis Sapient
1 week ago
Gurugram, India Publicis Sapient Full timeJob Description :As Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing Azure Cloud based big data solutions for clients. Your role will be focused on delivering high quality solutions by independently driving design discussions related to Data Ingestion, Transformation &...
-
Big Data Engineer
4 weeks ago
gurugram, India Intraedge Technologies Ltd. Full timeJob Description :We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.Key...
-
Big Data Engineer
1 week ago
Gurugram, India Intraedge Technologies Ltd. Full timeJob Description :We are seeking a skilled Big Data GCP Engineer to join our team. The ideal candidate will have substantial experience with Google Cloud Platform (GCP) and a strong background in Big Data technologies. This role involves working with various GCP services and tools to design, develop, and optimize data processing pipelines and workflows.Key...
-
Gurgaon/Gurugram, India Paisabazaar.com Full timeJob Description : Qualifications for Data Engineer : 3+ Years of experience in building and optimizing 'big data' solutions required to fulfill business and technology requirements. 3+ years of technical expertise in areas of design and implementation using big data technology Hadoop, Hive, Spark, Python/Java. Strong analytic skills to understand...
-
Gurgaon/Gurugram, India Paisabazaar.com Full timeJob Description : Qualifications for Data Engineer : 3+ Years of experience in building and optimizing 'big data' solutions required to fulfill business and technology requirements. 3+ years of technical expertise in areas of design and implementation using big data technology Hadoop, Hive, Spark, Python/Java. Strong analytic skills to understand...
-
Publicis Sapient
3 weeks ago
Gurugram, India Publicis Sapient Full timeJob Description :As Manager, Data Engineering, you will be responsible for translating client requirements into design, architecting, and implementing Azure Cloud based big data solutions for clients. Your role will be focused on delivering high quality solutions by independently driving design discussions related to Data Ingestion, Transformation &...
-
Big Data Developer
2 months ago
gurugram, India Rocket Consulting Full timeBig Data EngineerPosition Overview :We are seeking a skilled Big Data Engineer to join our dynamic team. The ideal candidate will have expertise in designing, implementing, and maintaining large-scale data processing systems using cutting-edge technologies. As a Big Data Engineer, you will play a key role in building and optimizing data pipelines, data...
-
Big Data Developer
2 months ago
Gurugram, India Rocket Consulting Full timeBig Data EngineerPosition Overview :We are seeking a skilled Big Data Engineer to join our dynamic team. The ideal candidate will have expertise in designing, implementing, and maintaining large-scale data processing systems using cutting-edge technologies. As a Big Data Engineer, you will play a key role in building and optimizing data pipelines, data...
-
Big Data Developer
2 weeks ago
Gurugram, India Shya Workforce Solutions Full timeCollaborate with enterprise architects, data architects, developers & engineers, data scientists, andinformation designers to lead the identification and definition of required data structures, formats,pipelines, metadata, and workload orchestration capabilitiesExperience in service architecture, development, and high performance and scalability.Experience...
-
Big Data Developer
2 weeks ago
Gurugram, India Shya Workforce Solutions Full timeCollaborate with enterprise architects, data architects, developers & engineers, data scientists, andinformation designers to lead the identification and definition of required data structures, formats,pipelines, metadata, and workload orchestration capabilitiesExperience in service architecture, development, and high performance and scalability.Experience...
-
Big Data Developer
2 weeks ago
gurugram, India Shya Workforce Solutions Full timeCollaborate with enterprise architects, data architects, developers & engineers, data scientists, and information designers to lead the identification and definition of required data structures, formats, pipelines, metadata, and workload orchestration capabilities Experience in service architecture, development, and high performance and scalability....
-
Senior Data Engineer
3 weeks ago
gurugram, India Optivalue tek consulting Full timeCompany Description : OptiValueTek is a leading provider of IT solutions specializing in Application & Process Integration, API Management, Data Engineering, Data Science/MLOps, DevOps & SRE, Cloud Solutions, and Web/Mobile & Application Development. With a focus on innovation, we have developed in-house tools/accelerators for Test Automation, API...
-
Big Data Engineer
1 week ago
Gurugram, India Virtusa Full timeBig Data Engineer - CREQ190572 Description Qualifications Bachelors degree in Engineering or Computer Science or equivalent OR Masters in Computer Applications or equivalent. 6+ years of software development experience and leading teams of engineers and scrum teams 3+ years of hands-on experience of working with Map-Reduce, Hive, Spark (core, SQL and...