Airflow Orchestration

**Vacancy No**

VN1197

**Business Unit**

EMEA

**Job Location**

India/Philippines

**Employment Type**

Full Time

**Job Details and Responsibilities**

**Key Responsibilities**:
**Workflow Migration**:

- Analyze and convert Oozie workflows into Airflow DAGs using Python-based orchestration.
- Design and implement reusable, modular, and optimized Airflow pipelines for data ingestion, transformation, and orchestration.
- Maintain a one-to-one mapping between legacy workflows and Airflow DAGs, ensuring no data loss or business interruption.

**Cloud Data Migration**:

- Collaborate with data engineering teams to migrate Cloudera Hadoop workloads to Databricks on AWS.
- Leverage Airflow for scheduling and orchestrating data workflows on AWS-based services (e.g., S3, EMR, Glue, Redshift).

**Pipeline Optimization**:

- Optimize data ingestion pipelines to achieve high throughput and low latency on AWS cloud infrastructure.
- Integrate Airflow workflows with Databricks for data transformations and analytics.

**Error Handling and Monitoring**:

- Implement robust error-handling mechanisms, task retries, and alerting within Airflow workflows.
- Set up monitoring dashboards using tools like CloudWatch, Prometheus, or Airflow’s built-in features.

**Collaboration and Documentation**:

- Work closely with data architects, cloud engineers, and DevOps teams to align on migration goals and architecture.
- Document the migration process, workflow logic, and best practices for ongoing maintenance.

**Performance Testing**:

- Conduct performance testing and benchmarking of migrated pipelines to ensure efficient resource utilization on Databricks and AWS.

**CI/CD Implementation**:

- Design and maintain CI/CD pipelines for orchestrated workflows using Jenkins and Terraform.
- Automate deployment of Airflow DAGs and infrastructure components using Terraform IaC (Infrastructure as Code).
- Implement quality checks and validation for workflow pipelines during the deployment process.

**Qualifications**

**Technical Expertise**:

- Strong experience in Apache Airflow, including designing and managing complex DAGs.
- Hands-on experience with Apache Oozie and migration to modern orchestration tools.
- Proficiency in Python for writing Airflow DAGs and custom operators/hooks.
- Experience with Hadoop ecosystems (Cloudera distribution preferred) and their components like HDFS, Hive, and Spar
- Solid experience with CI/CD tools, including Jenkins for pipeline automation and Terraform for data orchestration/pipeline provisioning.
- Familiarity with Databricks (on AWS preferred) and its integration with Airflow for ETL and data processing.

**Cloud and Infrastructure**:

- Solid understanding of AWS services such as S3, EMR, Glue, Lambda, Redshift, and IAM.
- Experience with containerization tools (Docker, Kubernetes) and CI/CD pipelines for workflow deployment.

**Analytical and Problem-Solving**:

- Ability to debug and resolve issues in data workflows and orchestrators.
- Experience optimizing workflows for performance and scalability.

**Preferred Qualifications**:

- Experience in large-scale cloud migrations, specifically from on-premises Hadoop to Databricks on AWS
- Knowledge of Spark and PySpark for big data transformations.
- Familiarity with version control tools (e.g., Git) and workflow monitoring tools.
- Certifications in AWS (e.g., AWS Certified Solutions Architect) or Databricks.
- Lemongrass Consulting is proud to be an Equal Opportunity and Affirmative Action employer. We do not discriminate on the basis of race, religion, color, national origin, religious creed, gender, sexual orientation, gender identity, gender expression, age, genetic information, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics._

**About Lemongrass**
- Lemongrass is a software-enabled services provider, synonymous with SAP on Cloud, focused on delivering superior, highly automated Managed Services to Enterprise customers. Our customers span multiple verticals and geographies across the Americas, EMEA and APAC. We partner with AWS, SAP, Microsoft and other global technology leaders.