Manager -Data Engineer (AWS, Python, Spark, Databricks for ETL - Agentic AI

Job Description

Inviting applications for the role of Manager -Data Engineer (AWS, Python, Spark, Databricks for ETL - Agentic AI

In this role, you%27ll be part of Genpact%27s transformation under GenpactNext, as we lead the shift to Agentic AI Solutions-domain-specific, autonomous systems that redefine how we deliver value to clients. You%27ll help drive the adoption of innovations like the Genpact AP Suite in finance and accounting, with more Agentic AI products set to expand across service lines.

Responsibilities

- Design, develop, and manage scalable ETL pipelines using AWS Glue, Databricks, Apache Spark, and Python to process structured and unstructured data from diverse sources.

- Manage releases, Oversee testing, PoCs , including cost evaluations, of various AWS services and other tools, AWS, Python, Spark, Databricks for ETL-Agentic AI, Strong experience in Python and SQL , Proven expertise in AWS services such as S3, Lambda, Glue, EMR, and Redshift. Strong understanding of security principles and best practices for cloud-based environments.

- Build and orchestrate data workflows integrating with services such as AWS Lambda, Step Functions, S3, and Redshift, ensuring high availability and performance.

- Optimize Spark jobs for performance and cost-efficiency across Databricks and AWS Glue environments using partitioning, job bookmarks, and dynamic frame operations.

- Maintain secure data solutions in AWS, leveraging IAM roles, KMS encryption, and VPC-based security to meet compliance and governance standards.

- Migrate legacy ETL jobs and data from on-prem systems to cloud-native architectures on AWS Glue, Redshift, and DynamoDB.

- Implement/monitor data pipeline performance, performing debugging and tuning of Spark jobs to ensure reliable execution and minimal downtime.

- Contribute in the design and review of technical solutions, translating business requirements and user stories into scalable data engineering architectures.

- Conduct unit testing and data validation to ensure functional correctness of pipelines before deployment.

- Contribute to production deployment and collaborate with release management to ensure seamless delivery of data solutions.

- Recommend cost-effective, secure, and high-performing cloud-based data solutions, reducing manual overhead and operational burden.

Qualifications we seek in you

Minimum Qualifications

- Experience in designing, implementing data pipelines, build data applications, data migration on AWS

- Strong experience of implementing data lake using AWS services like Glue, Lambda, Step, Redshift

- Experience of Databricks will be added advantage

- Strong experience in Python and SQL

- Proven expertise in AWS services such as S3, Lambda, Glue, EMR, and Redshift.

- Advanced programming skills in Python for data processing and automation.

- Hands-on experience with Apache Spark for large-scale data processing.

- Proficiency in SQL for data querying and transformation.

- Strong understanding of security principles and best practices for cloud-based environments.

- Experience with monitoring tools and implementing proactive measures to ensure system availability and performance.

- Excellent problem-solving skills and ability to troubleshoot complex issues in a distributed, cloud-based environment.

- Strong communication and collaboration skills to work effectively with cross-functional teams.

Preferred Qualifications/ Skills

- Bachelor's degree in business information systems (IS), computer science or related field, or equivalent-related IT experience.

- AWS Data Engineering & Cloud certifications, Databricks certifications

- Familiar with multiple data integration technologies and cloud platforms

- Knowledge of Change & Incident Management process

-

-

-

-

-

-