Data Engineer

The Data Engineer – MWAA & Redshift will design, build, and maintain data pipelines orchestrated through AWS Managed Workflows for Apache Airflow (MWAA) to ingest, transform, and optimize data in Amazon Redshift. This role is central to implementing automated, scalable, and reliable data workflows that support analytics, reporting, and downstream data products within a modern cloud data platform. About the Role Key Responsibilities Pipeline Design & Development Design, build, and maintain data ingestion and transformation pipelines using MWAA (Apache Airflow) and AWS Redshift. Develop Airflow DAGs to orchestrate data extraction from diverse sources (e.g., Oracle EBS, APIs, flat files, or S3). Implement transformation logic using SQL, Python, or Redshift stored procedures to curate data for analytical consumption. Create and manage staging, curated, and semantic data layers in Redshift aligned with the Medallion architecture principles. Integrate data ingestion pipelines with AWS services (e.g., S3, Glue, Lambda, Secrets Manager). Performance, Optimization & Maintenance Tune Redshift queries and schema for high performance and cost efficiency. Implement Airflow monitoring, logging, and alerting using CloudWatch or similar tools. Maintain and version-control Airflow DAGs and Redshift scripts in Git. Continuously improve data pipeline reliability, error handling, and observability. Establish data validation checks and ensure pipeline SLAs are met. Integration & Security Configure and manage secure data access between MWAA, S3, and Redshift using IAM roles and Secrets Manager. Collaborate with data architects and DevOps teams to implement CI/CD automation for pipeline deployment. Work closely with business analysts and data consumers to ensure datasets meet functional and performance requirements. Testing & Validation Perform unit, integration, and regression testing on data pipelines and transformations. Validate data quality, schema consistency, and referential integrity between source and Redshift layers. Participate in User Acceptance Testing (UAT) and resolve issues collaboratively with stakeholders. Document data flows, dependencies, and operational runbooks for pipeline management. Qualifications 5–8 years of experience in data engineering or related roles. Proven experience with Apache Airflow / MWAA for building and orchestrating data workflows. Strong proficiency in Amazon Redshift — including schema design, SQL optimization, and performance tuning. Solid understanding of AWS services: S3, Glue, Lambda, CloudWatch, Secrets Manager, and IAM. Experience with Python (for Airflow DAGs and data transformations). Strong SQL development skills for ELT logic in Redshift. Familiarity with Medallion architecture (Raw, Curated, Gold) or other layered data design patterns. Hands-on experience managing data ingestion from enterprise systems (e.g., Oracle EBS, SAP, Salesforce) is a plus. Understanding of data quality frameworks, testing automation, and operational monitoring. Required Skills 5–8 years of experience in data engineering or related roles. Proven experience with Apache Airflow / MWAA for building and orchestrating data workflows. Strong proficiency in Amazon Redshift — including schema design, SQL optimization, and performance tuning. Solid understanding of AWS services: S3, Glue, Lambda, CloudWatch, Secrets Manager, and IAM. Experience with Python (for Airflow DAGs and data transformations). Strong SQL development skills for ELT logic in Redshift. Familiarity with Medallion architecture (Raw, Curated, Gold) or other layered data design patterns. Hands-on experience managing data ingestion from enterprise systems (e.g., Oracle EBS, SAP, Salesforce) is a plus. Understanding of data quality frameworks, testing automation, and operational monitoring. Preferred Skills Hands-on experience managing data ingestion from enterprise systems (e.g., Oracle EBS, SAP, Salesforce) is a plus. Pay range and compensation package [Pay range or salary or compensation] Equal Opportunity Statement [Include a statement on commitment to diversity and inclusivity.]