Consultant -Data Engineer ( AWS, Python, Spark, Databricks for ETL -Agentic AI

Job Description

Inviting applications for the role of Consultant -Data Engineer ( AWS, Python, Spark, Databricks for ETL -Agentic AI

In this role, you%27ll be part of Genpact%27s transformation under GenpactNext, as we lead the shift to Agentic AI Solutions-domain-specific, autonomous systems that redefine how we deliver value to clients. You%27ll help drive the adoption of innovations like the Genpact AP Suite in finance and accounting, with more Agentic AI products set to expand across service lines.

Responsibilities

- Design, develop, and manage scalable ETL pipelines using AWS Glue, Databricks, Apache Spark, and Python to process structured and unstructured data from diverse sources.

- AWS, Python, Spark, Databricks for ETL-Agentic AI, Strong experience in Python and SQL , Proven expertise in AWS services such as S3, Lambda, Glue, EMR, and Redshift. Strong understanding of security principles and best practices for cloud-based environments. Write clean testable code, engage in code reviews and agile ceremonies

- Build and orchestrate data workflows integrating with services such as AWS Lambda, Step Functions, S3, and Redshift, ensuring high availability and performance.

- Optimize Spark jobs for performance and cost-efficiency across Databricks and AWS Glue environments using partitioning, job bookmarks, and dynamic frame operations.

- Develop and maintain secure data solutions in AWS, leveraging IAM roles, KMS encryption, and VPC-based security to meet compliance and governance standards.

- Migrate legacy ETL jobs and data from on-prem systems to cloud-native architectures on AWS Glue, Redshift, and DynamoDB.

- Implement and monitor data pipeline performance, performing debugging and tuning of Spark jobs to ensure reliable execution and minimal downtime.

- Collaborate in the design and review of technical solutions, translating business requirements and user stories into scalable data engineering architectures.

- Perform unit testing and data validation to ensure functional correctness of pipelines before deployment.

- Lead production deployment and coordinate with release management to ensure seamless delivery of data solutions.

- Recommend cost-effective, secure, and high-performing cloud-based data solutions, reducing manual overhead and operational burden.

- Contribute to backup, disaster recovery, and business continuity strategies for critical data assets.

- Participate in code reviews, technical design discussions, and DevOps integration for CI/CD of data pipelines using tools like Git, CodePipeline, or Databricks Repos.

Qualifications we seek in you

Minimum Qualifications

- Experience in designing, implementing data pipelines, build data applications, data migration on AWS

- Strong experience of implementing data lake using AWS services like Glue, Lambda, Step, Redshift

- Experience of Databricks will be added advantage

- Strong experience in Python and SQL

- Proven expertise in AWS services such as S3, Lambda, Glue, EMR, and Redshift.

- Advanced programming skills in Python for data processing and automation.

- Hands-on experience with Apache Spark for large-scale data processing.

- Proficiency in SQL for data querying and transformation.

- Strong understanding of security principles and best practices for cloud-based environments.

- Experience with monitoring tools and implementing proactive measures to ensure system availability and performance.

- Excellent problem-solving skills and ability to troubleshoot complex issues in a distributed, cloud-based environment.

- Strong communication and collaboration skills to work effectively with cross-functional teams.

Preferred Qualifications/ Skills

- Master's Degree-Computer Science, Electronics, Electrical or equivalent

- AWS Data Engineering & Cloud certifications, Databricks certifications

- Experience with multiple data integration technologies and cloud platforms

- Knowledge of Change & Incident Management process

-

-

-

-

-

-

---