AWS Data Engineer

Roles & Responsibilities
Role Summary
We are looking for a highly skilled
AWS Databricks Data Engineer
to design, develop, and optimize data pipelines and lakehouse solutions on AWS using Databricks. The ideal candidate will have strong experience in
PySpark, SQL, Delta Lake, AWS native services, Unity Catalog
and building scalable data processing solutions for analytics and AI/ML workloads.

Key Responsibilities

• Design, build, and optimize ETL/ELT pipelines using Databricks (PySpark, Delta Lake) on AWS.
• Ingest and process structured, semi-structured, and unstructured data from multiple sources (RDBMS, APIs, streaming).
• Integrate AWS services such as S3, Glue, Lambda, Kinesis, Athena, Redshift with Databricks.
• Implement data quality checks, validation frameworks, and monitoring for production workloads.
• Optimize data storage, partitioning, and caching strategies for performance and cost efficiency.
• Collaborate with Data Architects and Analysts to design data models and lakehouse layers (raw, curated, semantic).
• Implement CI/CD pipelines for Databricks jobs and notebooks using Git-based workflows (e.g., CodeCommit, GitHub).
• Apply security best practices: IAM roles, encryption, VPC, and data governance policies.
• Work with stakeholders to understand data requirements and deliver scalable, reusable solutions.
• Participate in code reviews, peer programming, and Agile ceremonies.

Required Skills & Experience

• 3–6 years of experience in Data Engineering (cloud-based preferred).
• Strong hands-on expertise in Databricks (PySpark, Delta Lake, SQL).
• Proficiency in AWS data services: S3, Glue, Lambda, Kinesis, Athena, Redshift.
• Experience in ETL/ELT design patterns and best practices.
• Strong Python and SQL skills for data manipulation and transformation.
• Knowledge of data warehousing, dimensional modeling, and lakehouse concepts.
• Experience with version control (Git) and DevOps tools.
• Understanding of data security, encryption, and governance on AWS.

Good To Have / Preferred

• AWS certifications: AWS Certified Data Engineer – Associate / Professional or AWS Certified Solutions Architect – Associate.
• DataBricks Certification: DataBricks certified Associate / Professional
• Experience with Glue or other orchestration tools.
• Exposure to machine learning pipelines in Databricks.
• Familiarity with Terraform or AWS CloudFormation for IaC.
• Experience in Agile/Scrum delivery models.

Education

• Bachelor's or Master's degree in Computer Science, Information Systems, or related field.

Soft Skills

• Strong problem-solving and analytical thinking.
• Excellent communication and collaboration skills.
• Ability to work independently and in cross-functional teams.
• Adaptability to changing priorities in a fast-paced environment.

Experience

• 6-8 Years

Skills

• Primary Skill: Data Engineering
• Sub Skill(s): Data Engineering
• Additional Skill(s): Python, AWS - CloudFormation, Data Modeling, Apache Hive, Azure Storage, BI & Visualization Development, Data Engineering, AWS-Apps, AWS-Infra, Azure Datalake, Azure Data Factory, Python (Data Science), AWS CloudTrail, Data Cloud

About The Company
Infogain is a human-centered digital platform and software engineering company based out of Silicon Valley. We engineer business outcomes for Fortune 500 companies and digital natives in the technology, healthcare, insurance, travel, telecom, and retail & CPG industries using technologies such as cloud, microservices, automation, IoT, and artificial intelligence. We accelerate experience-led transformation in the delivery of digital platforms. Infogain is also a Microsoft (NASDAQ: MSFT) Gold Partner and Azure Expert Managed Services Provider (MSP).

Infogain, an Apax Funds portfolio company, has offices in California, Washington, Texas, the UK, the UAE, and Singapore, with delivery centers in Seattle, Houston, Austin, Kraków, Noida, Gurgaon, Mumbai, Pune, and Bengaluru.