io - Lead Data Engineer - ETL/PySpark
2 weeks ago
Job Description :
We are looking for an experienced Lead Data Engineer to join our dynamic team and help us build innovative data engineering solutions that empower businesses to leverage the full potential of their data.
As a Lead Data Engineer, you will be responsible for building scalable data pipelines, managing large datasets, and designing end-to-end data architectures to derive actionable insights from terabyte-scale data.
Key Responsibilities :
- Build scalable data engineering solutions to digitize and derive insights from unused or underutilized data sources.
- Develop robust ETL processes (Extract, Transform, Load) that efficiently handle and transform large datasets, integrating them into a centralized data lake or warehouse.
- Create BI streaming pipelines to handle real-time data processing and provide actionable insights across business functions.
- Design and implement data solutions for terabyte-scale datasets, ensuring high performance, scalability, and reliability.
- Utilize cloud platforms such as Azure (Data Lakes, Data Factory, Databricks) and AWS (Snowflake) to architect cloud-based data solutions.
- Work with Big Data technologies such as Hadoop, PySpark, and Kafka to process large-scale data and ensure effective data storage and access.
- Manage end-to-end deployment of data pipelines and infrastructure using CI/CD pipelines, such as Jenkins, to streamline development, testing, and production deployment.
- Ensure automated testing, monitoring, and troubleshooting of data pipelines to guarantee continuous data flow and operational stability.
- Lead data engineering projects from inception to delivery, ensuring they meet business requirements, performance standards, and timelines.
- Work closely with international clients to understand their data requirements, deliver custom solutions, and provide expert advice on best practices.
- Collaborate with cross-functional teams, including data scientists, analysts, and business stakeholders, to design and implement data solutions.
- Mentor junior and mid-level engineers, helping them to improve their technical skills and grow professionally within the company.
- Drive adoption of best practices in data engineering, including data governance, security, and compliance with industry standards.
- Foster a culture of continuous learning and innovation within the data engineering team.
Required Skills & Qualifications :
- Expertise in Azure (especially Azure Data Lake, Data Factory, Databricks) and/or AWS (particularly Snowflake).
- Hands-on experience in building cloud-based data architectures and scalable data pipelines.
- Proficiency in Hadoop, Kafka, PySpark, and SQL to process and manipulate large datasets.
- Strong experience working with data lakes, data warehouses, and real-time data streaming.
- Strong programming skills in Python and PySpark for data manipulation and transformation.
- Extensive experience writing optimized SQL queries for complex data operations.
- 8-12 years of experience in Data Engineering with a focus on Big Data and cloud solutions.
- Proven ability to lead teams and manage end-to-end project delivery while working with cross-functional teams and international clients.
- Experience with CI/CD pipelines, particularly in deploying and managing data engineering solutions using tools like Jenkins.
- Strong understanding of data architecture, ETL processes, data lakes, and data warehousing concepts.
- Ability to design solutions for both batch and streaming data
-
Pyspark Data Engineer
2 weeks ago
Chennai, Tamil Nadu, India Synechron Full timeAbout the Role: We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role...
-
Pyspark Data Engineer
3 weeks ago
Chennai, Tamil Nadu, India Synechron Full timeAbout the Role:We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that ensure high data quality and availability across the organization. This role...
-
PySpark Developer Lead
1 day ago
Chennai, Tamil Nadu, India O2F Info Solutions Pvt. Ltd. Full timeRequired Skills and Qualifications:Experience: 4-8 years of experience in data engineering, with a strong focus on PySpark and large-scale data processing.Technical Skills: Expertise in PySpark for distributed data processing, data transformation, and job optimization. Strong proficiency in Python and SQL for data manipulation and pipeline creation.Hands-on...
-
Pyspark Developer
1 week ago
Chennai, Tamil Nadu, India MP DOMINIC AND CO Full timeJob Summary - Design develop and implement scalable data pipelines and streaming use cases using PySpark and Spark on a distributed computing platform - Possess strong programming skills in Spark streaming - Have familiarity with cloud platforms like GCP - Gain experience in big data technologies such as Hadoop Hive and HDFS - Perform ETL operations...
-
Lead Data Engineer
2 days ago
Chennai, Tamil Nadu, India Cynosure Corporate Solutions Full timeJob DescriptionCynosure Corporate Solutions is seeking a highly skilled Senior Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for designing and implementing scalable data pipelines using Spark/Pyspark with SQL and Python or SCALA.Key Responsibilities:• Participate in system design meetings and collect...
-
Data Engineer
2 days ago
Chennai, Tamil Nadu, India O2F Info Solutions Pvt. Ltd. Full timeJob Summary : We are seeking a highly skilled Senior Data Engineer with 4 to 8 years of experience in building robust data pipelines and working extensively with PySpark to join our data engineering team. Key Responsibilities : Data Pipeline Development : - Design, build, and maintain scalable data pipelines using PySpark to process large datasets and...
-
Data Engineer
2 days ago
Chennai, Tamil Nadu, India Xerago Full timeWe are looking for an experienced Python ETL Developer to design, develop, and optimize data pipelines. The ideal candidate should have expertise in Python, PySpark, Airflow, and data processing frameworks, along with the ability to work independently and communicate effectively in English. Roles & Responsibilities : - Develop and maintain ETL pipelines...
-
Data Engineer Lead
2 days ago
Chennai, Tamil Nadu, India METRIXIT SOLUTIONS PVT LTD Full timeAbout the Job:We are seeking a highly skilled Data Engineer Lead to join our team at Metrixit Solutions Pvt Ltd. As a key member of our data engineering group, you will be responsible for designing, developing, and maintaining scalable, efficient, and high-performance data pipelines for ETL processes.Key Responsibilities:Design and develop data pipelines...
-
Data Engineer
2 weeks ago
Chennai, Tamil Nadu, India Xerago Full timeWe are looking for an experienced Python ETL Developer to design, develop, and optimize data pipelines. The ideal candidate should have expertise in Python, PySpark, Airflow, and data processing frameworks, along with the ability to work independently and communicate effectively in English.Roles & Responsibilities : - Develop and maintain ETL pipelines using...
-
Pyspark Architect
2 weeks ago
Chennai, Tamil Nadu, India Saaki Argus & Averil Consulting Full timeAbout Company: A global IT services and consulting company specializing in digital transformation, cloud services, and automation solutions. Role: Pyspark Architect Experience: 15 -22 Years Location: Mumbai, Pune, Chennai, Bangalore & Noida. Job Description:Proven experience as a Technical Architect or similar role with a strong focus on Python...
-
Experienced ETL Developer
2 days ago
Chennai, Tamil Nadu, India Xerago Full timeXerago is looking for an Experienced ETL Developer to lead our data pipeline optimization efforts.About the RoleDevelop and maintain complex ETL pipelines using Python, NumPy, Pandas, PySpark, and Apache Airflow.Collaborate with cross-functional teams to ensure efficient data processing and transformation workflows.Optimize ETL performance and scalability...
-
Data Engineer
3 weeks ago
Chennai, Tamil Nadu, India Synechron Full timeJob Description: We are seeking a highly skilled Data Engineer with deep expertise in PySpark and the Cloudera Data Platform (CDP) to join our data engineering team. Responsibilities: Design, develop, and maintain highly scalable and optimized ETL pipelines using PySpark on the Cloudera Data Platform, ensuring data integrity and accuracy. Implement and...
-
AWS Data Engineer(Chennai, Indore
2 weeks ago
Chennai, Tamil Nadu, India Tata Consultancy Services Full timeRole: AWS Data EngineerJOB LOCATION : Chennai, Indore , PuneEXPERIENCE REQUIREMENT : 4+Required Technical Skill:Strong Knowledge of Aws Glue/AWS REDSHIFT/SQL/ETL. Good knowledge and experience in Pyspark for forming complex Transformation logic.AWS Data Engineer,SQL,ETL, DWH , Secondary : AWS Glue , AirflowMust-HaveGood Knowledge of SQL , ETLA minimum of 3 +...
-
Data Engineering Lifecycle Manager
7 days ago
Chennai, Tamil Nadu, India Thryve Digital Health LLP Full timeJob Role:As a seasoned Data Engineering Lifecycle Manager, you will lead a vibrant data team in creating and maintaining reusable data pipelines using Databricks (PySpark) and GCP. Your role will involve managing the work end-to-end, interacting with multiple stakeholders, including US counterparts and other vendor team members.About the Role:You will be...
-
Senior Data Engineering Manager
2 days ago
Chennai, Tamil Nadu, India Thryve Digital Health LLP Full timeJob Description:As a Senior Data Engineering Manager at Thryve Digital Health LLP, you will lead a high-performing team of data engineers in designing and implementing end-to-end data pipelines using Databricks (PySpark) and GCP. Your expertise in managing complex data projects and programs will be crucial in ensuring seamless integration across upstream and...
-
gcp data engineer lead
4 weeks ago
Chennai, Tamil Nadu, India Mastech Digital Full timePosition: GCP Data Engineer Lead Location: Chennai, Tamil Nadu(3 day/week) / (Remote with travel 1 week/ 3 months) Duration: Full Time Notice Period: 15-30 days(Non negotiable) Service based Company: Mastech InfoTrellis and Mastech Digital Company Name: Mastech Digital and Mastech InfoTrellis Company Link- Job Description: We are seeking a...
-
Chennai, Tamil Nadu, India Tata Consultancy Services Full timeRole: AWS Data EngineerJOB LOCATION : Chennai, Bangalore, Mumbai, Indore ,PuneEXPERIENCE REQUIREMENT : 4+Required Technical Skill:Strong Knowledge ofAws Glue/AWS REDSHIFT/SQL/ETL.Good knowledge and experience in Pyspark for forming complex Transformation logic.AWS Data Engineer,SQL,ETL,DWH, Secondary : AWS Glue , AirflowMust-HaveGood Knowledge of SQL , ETLA...
-
Chennai, Tamil Nadu, India Nowwin International Pvt Ltd Full timeWe hiring ADF Developer for our contract project. Experience : 4 to 6 years. Location : Chennai Onsite Must. Contract Duration : 3 months. Must Skills : ADF, SQL and ETL. Role : ADF. Roles & Responsibilities : - Develop, implement, and optimize Azure Data Factory (ADF) pipelines for data extraction, transformation, and loading (ETL). - Work with Azure...
-
Data Engineer
3 weeks ago
Chennai, Tamil Nadu, India Synechron Full timeGreetings,We have an urgent opening for a Data Engineer specializing in PySpark at Synechron in Chennai. We are looking for candidates with more than 5+ years of relevant experience.Position: Data Engineer-PysparkLocation: ChennaiAbout Company:At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm...
-
Data Engineer
2 weeks ago
Chennai, Tamil Nadu, India Synechron Full timeGreetings,We have an urgent opening for a Data Engineer specializing in PySpark at Synechron in Chennai. We are looking for candidates with more than 5+ years of relevant experience.Position: Data Engineer-PysparkLocation: ChennaiAbout Company:At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm...