Senior Data Orchestration Specialist

2 days ago

Remote, India beBeeDataOrchestration Full time

We are seeking a seasoned Data Orchestration Engineer to lead the migration of legacy data workflows from Apache Oozie to Apache Airflow.

This is a critical role focused on reengineering data pipeline orchestration, automation, and deployment within a cloud-native framework.

  • Convert Oozie workflows to Airflow DAGs using Python.
  • Build reusable, modular Airflow pipelines for ingestion, transformation, and scheduling.
  • Ensure accurate migration and one-to-one workflow alignment without disrupting business processes.
Cloud Data Platform Transition
  • Work with engineering teams to migrate Hadoop workloads to Databricks on AWS.
  • Leverage Airflow to orchestrate data pipelines across AWS services (S3, EMR, Glue, Redshift).
Pipeline Optimization
  • Enhance pipeline performance for throughput and latency in the AWS ecosystem.
  • Integrate Airflow with Databricks for transformation and analytics tasks.
Monitoring & Error Handling
  • Implement retry logic, exception handling, and alerting in Airflow.
  • Set up observability tools like CloudWatch, Prometheus, or Airflow's native monitoring.
Collaboration & Documentation
  • Collaborate with data architects, DevOps, and cloud teams.
  • Document orchestration logic, best practices, and the migration process.
CI/CD and Infrastructure Automation
  • Develop CI/CD pipelines using Jenkins and Terraform.
  • Automate DAG deployment and infrastructure provisioning via IaC.
  • Integrate validation steps in deployment workflows.

The ideal candidate will have strong expertise in Apache Airflow, including complex DAG design and orchestration. Prior experience with Apache Oozie and workflow migration is also required.

Additionally, proficiency in Python for Airflow DAG development and hands-on experience with Hadoop ecosystems are necessary. The candidate should also be familiar with CI/CD tools such as Jenkins and Infrastructure as Code (IaC) with Terraform.

Experience with Databricks (preferably on AWS) and big data orchestration is highly desirable. A solid understanding of AWS services: S3, EMR, Glue, Lambda, Redshift, IAM is also essential.

Familiarity with container tools such as Docker or Kubernetes is a plus. AWS or Databricks certifications are also beneficial.

  • Data Workflow Optimization Expert

    2 days ago

    Remote, India beBeeOrchestration Full time

    Job TitleApache Airflow Orchestration SpecialistWe are seeking an experienced and highly skilled Apache Airflow Orchestration Specialist to play a pivotal role in our enterprise data modernization programs. This role will focus on migrating legacy workflows from Oozie to Apache Airflow, orchestrating data pipelines on AWS, and optimizing data ingestion and...

  • Chief Data Architecture Specialist

    1 day ago

    Remote, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    **Job Description:**As a data engineer, you will be responsible for designing, maintaining and orchestrating data pipelines in an AWS environment for production processing and training flows.You will continuously evaluate, analyze, test and improve the quality, privacy and performance of our data systems. Additionally, you will contribute across the product,...

  • Data Insights Specialist

    3 days ago

    Remote, India beBeeData Full time ₹ 6,00,000 - ₹ 12,00,000

    Job Title: Data Insights SpecialistWe are seeking an experienced data insights specialist to join our team. In this role, you will be responsible for analyzing and interpreting complex data sets to inform business decisions.

  • Data Integration Specialist

    3 days ago

    Remote, India beBeeDataIntegration Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title: Data Integration SpecialistWe are seeking a highly skilled Data Integration Specialist to join our team.Our ideal candidate will have excellent development skills in PySpark, Python 2.x or 3.x.They should also possess excellent development skills in SQL, PL/SQL (Oracle, Teradata, Sybase).Familiarity with working in Unix/Linux environment is...

  • Backend Tester with SQL

    17 hours ago

    Remote, India Data PlatformExperts Full time

    Job Description: Backend Testing Specialist with Strong SQL Skills Position Overview: We are seeking a highly skilled and detail-oriented Backend Testing Specialist with strong SQL skills to join our team. As a Backend Testing Specialist, you will be responsible for ensuring the quality and reliability of our backend systems through rigorous testing and...

  • Senior Data Migration Specialist

    3 days ago

    Remote, India beBeeDataMigration Full time ₹ 5,00,000 - ₹ 10,00,000

    Data Migration ExpertThis is a senior-level position within the team responsible for managing large-scale data migrations. The ideal candidate will possess excellent technical skills and have prior experience with ETL tools.Key Responsibilities:Lead data analysis, mapping, and transfer processes to ensure seamless migration of data from one system to...

  • Cloud Infrastructure Specialist

    2 days ago

    Remote, India beBeeInfrastructure Full time

    As a Cloud Infrastructure Specialist, you will design and deploy scalable, secure, and cost-efficient infrastructure on AWS using Terraform.Key ResponsibilitiesDesign and provision scalable, secure, and cost-efficient infrastructure on AWS using Terraform (mandatory).Deploy and manage services such as Databricks, Amazon Redshift, EMR, and RDS using...

  • Senior Business Intelligence Migration Strategist

    3 days ago

    Remote, India beBeeDataMigration Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title: Senior Data Migration SpecialistAbout the RoleWe are seeking a highly skilled Senior Data Migration Specialist to lead our data migration programme. The successful candidate will be responsible for overseeing and ensuring delivery of all aspects of data analysis, mapping, and transfer.Key ResponsibilitiesTo form part of a team responsible for...

  • Senior Data Solutions Designer

    3 days ago

    Remote, India beBeeDatabase Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title: Senior Data Solutions DesignerJob DescriptionWe are seeking a highly skilled Senior Data Solutions Designer to join our team. As a key member of our data management team, you will be responsible for designing and implementing effective database solutions and models to store and retrieve company data.The ideal candidate will have a deep...

  • Data Systems Specialist

    3 days ago

    Remote, India beBeeDataSystemsSpecialist Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Description:We are seeking an experienced Data Systems Specialist to join our team. In this role, you will be responsible for designing, implementing, and maintaining data pipelines using AWS services such as Glue, EMR, S3, Redshift, RDS, and Airflow.The ideal candidate will have a strong background in programming using Python, Scala, or Spark. You...