Emergys - Big Data Engineer - ETL/Python
3 weeks ago
About the Role :
We are seeking a highly skilled and experienced Big Data Engineer to join our team in Pune. In this role, you will be responsible for designing, developing, and maintaining scalable big data solutions using Spark, PySpark, and related technologies.
You will work on building and optimizing data pipelines, implementing ETL processes, and ensuring data quality and reliability.
Responsibilities :
Spark Development :
- Design, develop, and optimize Spark applications for batch and streaming data processing.
- Implement data transformations and aggregations using PySpark.
- Develop and maintain data pipelines for real-time and batch processing.
- Optimize Spark performance for large datasets.
PySpark and Data Engineering :
- Utilize PySpark for data processing, transformation, and analysis.
- Implement data engineering best practices for data quality and reliability.
- Design and implement data models for big data solutions.
ETL Implementation and Migration :
- Implement and migrate ETL processes to Spark using PySpark.
- Develop and maintain data integration workflows.
- Optimize data loading and extraction processes.
Streaming Data Processing :
- Develop and implement real-time data streaming solutions using Kafka and Spark Streaming (DStreams and Structured Streaming).
- Design and implement data processing pipelines for streaming data.
- Troubleshoot and resolve issues related to streaming data processing.
Workflow Automation :
- Design and implement data workflows using Airflow or other workflow engines.
- Automate data processing tasks and workflows.
- Monitor and maintain data workflows.
Development Tools and Environments :
- Utilize Jupyter notebooks or other developer tools for data exploration and development.
- Implement and maintain development environments for big data solutions.
Programming and Scripting :
- Develop and maintain code in Python, Scala, and Java.
- Write efficient and maintainable code.
Required Technical Skills :
Spark :
- Extensive experience with Spark development (batch and streaming).
- Proficient in PySpark.
Streaming :
- Experience with Kafka and Spark Streaming (DStreams and Structured Streaming).
Programming :
- Strong programming skills in Python, Scala, and Java.
ETL :
- Experience with ETL implementation and migration to Spark.
Workflow Engines :
- Experience with Airflow or other workflow engines.
Development Tools :
- Experience with Jupyter notebooks or other developer tools.
Data Engineering :
- Strong understanding of data engineering principles and best practices.
Good to Have Skills :
Streaming Technologies :
- Experience with Flink and Kudu streaming.
- Experience with Nifi streaming and transformations.
Automation and CI/CD :
- Experience with automation of workflows and CI/CD pipelines.
Migration :
- Experience with Informatica workflow migration.
Required Experience :
- 4-7 years of experience as a Big Data Engineer.
- Proven experience in Spark development and data engineering.
- Experience with real-time data streaming and ETL processes.
Soft Skills :
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration skills.
- Ability to work independently and as part of a team.
- Strong attention to detail.
- Ability to learn and adapt to new technologies.
Education : Bachelor's degree in Computer Science, Information Technology, or a related field
-
Emergys - Data Scientist - NLP
2 weeks ago
Pune, Maharashtra, India Emergys labs Full timeYou will play a crucial role in developing and implementing cutting-edge AI/ML solutions, leveraging your expertise in generative models, NLP, and deep learning to solve complex business challenges. This role requires a deep understanding of advanced statistical and machine learning techniques, excellent communication skills, and the ability to collaborate...
-
Emergys - Data Scientist - NLP
4 weeks ago
Pune, Maharashtra, India Emergys labs Full timeYou will play a crucial role in developing and implementing cutting-edge AI/ML solutions, leveraging your expertise in generative models, NLP, and deep learning to solve complex business challenges. This role requires a deep understanding of advanced statistical and machine learning techniques, excellent communication skills, and the ability to collaborate...
-
Emergys - AWS Architect
2 weeks ago
Pune, Maharashtra, India Emergys labs Full timeKey Responsibilities : AWS Architecture & Application Development : - Design and implement scalable, secure, and high-performing AWS-based applications. - Develop cloud-native applications using Amplify, AWS Lambda, API Gateway, ECS, EKS, Fargate, and Step Functions. - Architect and manage serverless and containerized applications. - Lead the modernization...
-
Big data developer
5 days ago
Pune, Maharashtra, India LTIMindtree Full timeRole: Big Data EngineerExperience YearsMandatory Skills: Big Data and Pyspark, Core Spark, PythonJob Description:Relevant Experience in ETL and Data EngineeringStrong Knowledge in Spark, PythonStrong experience in Hive/SQL, PL/SQLGood Understanding of ETL & DW Concepts, Unix ScriptingDesign, implement and maintain Dat Pipeline to meet business...
-
Pune, Maharashtra, India Creospan Private Limited Full timeJob Title : Senior ETL Developer (Informatica & Big Data). Experience : 5-7 Years. Location : Pune, India. Company : Creospan. About the Role : Creospan is seeking a skilled Senior ETL Developer with 5-7 years of experience in Informatica Big Data Developer and advanced data integration technologies. This role focuses on developing, optimizing, and managing...
-
Big Data Developer
2 weeks ago
Pune, Maharashtra, India LTIMindtree Full timeRole: Big Data EngineerExperience - 8 - 12 YearsMandatory Skills: Big Data and Pyspark, Core Spark, PythonJob Description:Relevant Experience in ETL and Data EngineeringStrong Knowledge in Spark, PythonStrong experience in Hive/SQL, PL/SQLGood Understanding of ETL & DW Concepts, Unix ScriptingDesign, implement and maintain Dat Pipeline to meet business...
-
Big Data Developer
4 weeks ago
Pune, Maharashtra, India LTIMindtree Full timeRole: Big Data Engineer Experience - 3 - 8 Years Location - Chennai and PuneMandatory Skills:Big Data and PysparkJob Description: Relevant Experience in ETL and Data Engineering Strong Knowledge in Spark, Python Strong experience in Hive/SQL, PL/SQL Good Understanding of ETL & DW Concepts, Unix Scripting Design, implement and maintain Dat Pipeline to meet...
-
Big Data Developer
2 weeks ago
Pune, Maharashtra, India LTIMindtree Full timeRole: Big Data Engineer Experience - 3 - 8 Years Location - Chennai and Pune Mandatory Skills: Big Data and Pyspark Job Description: Relevant Experience in ETL and Data Engineering Strong Knowledge in Spark, Python Strong experience in Hive/SQL, PL/SQLGood Understanding of ETL & DW Concepts, Unix Scripting Design, implement and maintain Dat Pipeline to meet...
-
Senior Big Data Engineer
7 days ago
Pune, Maharashtra, India HCLTech Full timeJob Summary:HCLTech is seeking a Senior Big Data Engineer with 10+ years of experience in HDF5, Hive, Impala, and HBase, specializing in database management, data migration, and ETL processes within the banking domain. The ideal candidate will play a key role in designing, developing, and optimizing large-scale data solutions while ensuring compliance with...
-
Big Data Developer
5 days ago
Pune, Maharashtra, India LTIMindtree Full timeRole: Big Data EngineerExperience - 8 - 12 YearsMandatory Skills:Big Data and Pyspark, Core Spark, PythonJob Description:Relevant Experience in ETL and Data EngineeringStrong Knowledge in Spark, PythonStrong experience in Hive/SQL, PL/SQLGood Understanding of ETL & DW Concepts, Unix ScriptingDesign, implement and maintain Dat Pipeline to meet business...
-
Big Data Engineer with Amazon Redshift
5 days ago
Pune, Maharashtra, India BSC Services Pvt. Ltd. Full timeWe are a leading IT consultancy firm, BSC Services Pvt. Ltd., seeking an experienced Big Data Engineer to join our team.Job SummaryThe successful candidate will be responsible for designing, developing, and implementing a scalable big data architecture using Amazon Red Shift, DBT, and Airflow.Key ResponsibilitiesDesign and develop a robust data ingestion...
-
Big Data Engineer Lead
7 days ago
Pune, Maharashtra, India TIAA Full timeAbout the RoleAs a Data Engineer/Big Data Engineer at TIAA, you will have the opportunity to design and build cutting-edge data systems and architectures that drive business outcomes. You will collaborate with cross-functional teams to develop technical solutions that meet business needs and stay up-to-date with the latest trends and advancements in big...
-
Big Data Developer
1 week ago
Pune, Maharashtra, India Creospan Private Limited Full time**About Us**Creospan Private Limited is a leading provider of data solutions that seeks to revolutionize the way businesses operate. Our mission is to empower organizations to make informed decisions through our cutting-edge data analytics services.We are currently seeking an experienced Senior ETL Developer to join our team. As a key member of our team, you...
-
Big Data Engineering Lead
1 week ago
Pune, Maharashtra, India RefRelay Full timeJob DescriptionWe are RefRelay, a leading innovator in data engineering. We're seeking an experienced Data Engineer & QA to design, develop, and maintain scalable ETL pipelines and data infrastructure.Key Responsibilities:Design and implement robust ETL pipelines to ensure seamless data flow.Collaborate with cross-functional teams to define data quality...
-
Emergys - Hadoop Administrator - On-Premises
2 weeks ago
Pune, Maharashtra, India Emergys labs Full timeAbout the Role : We are seeking a highly skilled and experienced Hadoop Administrator to join our team. In this role, you will be responsible for the setup, configuration, maintenance, and security of our Hadoop clusters, both on-premises and in the cloud. You will leverage your deep understanding of Hadoop distributions and Unix-based operating systems to...
-
Big Data Engineer
1 day ago
Pune, Maharashtra, India Talent Corner Hr Services Private Limited Full timeJob DescriptionJob descriptionDesign, develop, maintain efficient data processing pipelines using PySpark.Implement best practices for ETL processes, ensuring high-quality & secure data.Monitor, troubleshoot, resolve issues related to data pipelines & infrastructure.Required Candidate profileExp in PySpark & Python.Exp with big data frameworks like Hadoop,...
-
Big Data Architect
1 day ago
Pune, Maharashtra, India TIAA Full timeAbout Our TeamWe are a dynamic team of professionals dedicated to delivering innovative solutions in data engineering and analytics. We strive to stay ahead of the curve by embracing new technologies and methodologies, and we encourage collaboration and knowledge-sharing among team members.Job SummaryWe are seeking an experienced Data Systems Engineer/Big...
-
Lead Big Data Engineer
1 week ago
Pune, Maharashtra, India Luxoft Full timeJob Title: Big Data Lead Engineer Description: We are seeking a seasoned Big Data Lead Engineer to lead the development and implementation of our cloud-based data processing platform. The ideal candidate will have expertise in designing and building scalable, reliable, and secure data pipelines using Azure, Apache Spark, and Data Lakehouse architecture. ...
-
Big Data Engineer Specialist
4 days ago
Pune, Maharashtra, India Undisclosed Full timeJob DescriptionThe successful candidate shall be placed at any of our Client's locations in Pune, Maharashtra. This is a Full-time Job with no remote work.Data Engineers willing to work on short-term contract or full-time in building ETL pipelines with strong experience in SQL for building queries, proficiency in Python, Java or Scala are preferred.Hands on...
-
Pune, Maharashtra, India Emergys labs Full timeAzure DevOps Engineer Position at Emergys LabsWe are seeking an experienced Senior Azure DevOps Engineer to join our growing cloud engineering team. As a key member of our team, you will be responsible for designing, implementing, and maintaining our Azure DevOps infrastructure and CI/CD pipelines.Key ResponsibilitiesDesign and implement Azure DevOps...