Data Engineer – AWS Big Data

Join a team focused on building large-scale data platforms and analytics solutions. As a Data Engineer, you will design, develop, and optimize high-volume data pipelines leveraging AWS cloud services, PySpark, and modern data lake architectures. Your Role You will be responsible for developing and maintaining scalable data pipelines, processing large datasets, and enabling reliable data solutions for business analytics and operational reporting. The role involves working with AWS services, ETL/ELT frameworks, orchestration tools, and data lake technologies. In This Role, You Will:
• Design, develop, and maintain scalable data pipelines to process data from multiple enterprise platforms.
• Build, manage, and optimize large-scale ETL/ELT workflows handling high-volume data workloads.
• Develop data processing applications using PySpark, Spark SQL, and Python on AWS EMR.
• Create and optimize Athena (Trino/Presto) queries for reporting, troubleshooting, and data analysis.
• Design, implement, and enhance data lake solutions using Amazon S3, AWS Glue, Athena, and EMR.
• Collaborate with business analysts, data scientists, and stakeholders to deliver reliable and scalable data solutions.
• Support migration and modernization initiatives involving Airflow/MWAA and workflow orchestration platforms.
• Implement data validation, monitoring, and quality checks to ensure data accuracy and consistency.
• Troubleshoot production pipeline failures, performance bottlenecks, and data quality issues.
• Follow DevOps and CI/CD best practices using GitLab for code management and deployments.
• Participate in architecture discussions, code reviews, and technical design activities.
• Create technical documentation, test plans, and operational runbooks.

Your Profile
Mandatory Skills
• 6–12 years of experience in Data Engineering and Big Data technologies.
• Strong hands-on experience with Python, PySpark, and Spark SQL.
• Expertise in AWS EMR, Amazon S3, AWS Glue, and Athena.
• Experience building and maintaining large-scale ETL/ELT pipelines.
• Strong knowledge of data lake architectures and data warehousing concepts.
• Experience with workflow orchestration tools such as Airflow/MWAA.
• Strong SQL skills and experience with Trino/Presto/Athena query optimization.
• Experience with data validation, monitoring, and troubleshooting.
• Familiarity with GitLab, CI/CD pipelines, and DevOps practices.
• Strong analytical and problem-solving capabilities. Preferred Skills
• Experience with telecom, network, billing, provisioning, or large enterprise datasets.
• Knowledge of real-time and batch data processing frameworks.
• Exposure to data governance and data quality frameworks.
• Experience with cloud-native analytics and reporting solutions.
• Understanding of Agile development methodologies.
• What You'll Love About Working Here
• Work on large-scale cloud and big data platforms.
• Exposure to modern AWS data engineering technologies.
• Opportunity to solve complex data processing and analytics challenges.
• Collaborative environment with data scientists, engineers, and business teams.
• Continuous learning and career growth in cloud and data engineering. About Capgemini in India Capgemini is an AI-powered global business and technology transformation partner, delivering tangible business value. We imagine the future of organizations and make it real with AI, technology and people. With our strong heritage of nearly 60 years, we are a responsible and diverse group of 420,000 team members in more than 50 countries. We deliver end-to-end services and solutions with our deep industry expertise and strong partner ecosystem, leveraging our capabilities across strategy, technology, design, engineering and business operations. The Group reported 2024 global revenues of €22.1 billion. Capgemini in India comprises nearly 235,000 team members working across 13 locations: Bangalore, Bhubaneswar, Chennai, Coimbatore, Gandhinagar, Gurugram, Hyderabad, Kolkata, Mumbai, Noida, Pune, Salem, and Tiruchirappalli. Make it real | www.capgemini.com/in-en.