Senior Pipeline Engineer

5 days ago

India Weekday AI Full time ₹ 20,00,000 - ₹ 25,00,000 per year

This role is for one of Weekday's clients
Min Experience: 6 years
Location: India
JobType: full-time

We are looking for an experienced and motivated Senior Pipeline Engineer to join our engineering team and take ownership of designing, building, and managing high-performance data pipelines. This role demands strong expertise in orchestration frameworks such as Celery and Temporal, combined with advanced skills in database design and query optimization. The ideal candidate will have a passion for solving complex technical challenges and delivering reliable, scalable solutions that power data-driven decision-making across the organization.

As a senior team member, you will play a critical role in shaping the future of our data infrastructure, working closely with cross-functional teams including data scientists, backend engineers, and product managers. You will also mentor junior engineers and drive best practices in orchestration, database efficiency, and scalable pipeline design.

Key Responsibilities
  • Pipeline Development & Management
    • Design, develop, and maintain robust and scalable data pipelines for batch and real-time processing.
    • Automate workflows for data ingestion, transformation, validation, and delivery.
    • Ensure pipelines are fault-tolerant, performant, and cost-efficient.
  • Workflow Orchestration
    • Build and manage orchestration workflows using Celery, Temporal, or similar tools.
    • Monitor, schedule, and optimize pipeline execution to reduce latency and improve reliability.
    • Implement scalable solutions for dependency management, retries, and error handling.
  • Database Design & Query Optimization
    • Create efficient database schemas and models to support large-scale data operations.
    • Optimize complex queries, indexing, and partitioning strategies to ensure high performance.
    • Collaborate with engineering teams to enforce database best practices and scalability standards.
  • Collaboration & Leadership
    • Partner with product managers, analysts, and business stakeholders to understand requirements and translate them into technical solutions.
    • Mentor and guide junior engineers in pipeline development and orchestration practices.
    • Contribute to architecture reviews, design discussions, and technical documentation.
  • Continuous Improvement & Compliance
    • Proactively identify opportunities to improve pipeline performance and reliability.
    • Stay updated on industry trends, orchestration frameworks, and database technologies.
    • Ensure adherence to data security, governance, and compliance policies.
Required Skills & Qualifications
  • 6+ years of experience in pipeline engineering, data engineering, or related fields.
  • Strong hands-on expertise in Celery, Temporal, or other orchestration frameworks.
  • Proficiency in database design, query optimization, and performance tuning.
  • Solid programming background in Python, Java, or similar languages.
  • Deep understanding of distributed systems, messaging queues, and event-driven architectures.
  • Experience with cloud platforms (AWS, Azure, GCP) and containerized environments (Docker, Kubernetes).
  • Strong problem-solving, analytical, and debugging skills.
  • Excellent communication and collaboration abilities.
Preferred Skills
  • Familiarity with real-time data streaming platforms like Kafka or Pulsar.
  • Experience with data warehousing tools (Snowflake, BigQuery, Redshift).
  • Knowledge of CI/CD pipelines, DevOps practices, and infrastructure automation.

  • Lead Data Pipeline Developer

    2 weeks ago

    India beBeeData Full time ₹ 6,00,000 - ₹ 8,00,000

    Senior Web Engineer Job DescriptionWe are seeking an experienced Senior Web Engineer to build a high-throughput product data ingestion pipeline across hundreds of domains. The successful candidate will design, develop, and implement scalable web crawlers using Python.The role involves collaborating with the data team to integrate vendor/public APIs...

  • Offshore Expert

    2 weeks ago

    India beBeeSeniorFieldEngineer Full time US$ 80,000 - US$ 1,20,000

    Job SummaryA highly skilled Senior Field Engineer is required to lead offshore transportation and installation projects for structures and pipelines. The ideal candidate will have a strong background in this field with at least three to five years of experience.Main Responsibilities:Oversee EPCIC (Engineering, Procurement, Construction, Installation, and...

  • Data pipeline engineer

    8 hours ago

    India JRD Systems Full time

    The Role: Backend and Data Pipeline Engineer:The Team: We are investing in technology to develop new products that help our customers drive their growth and transformation agenda. These include new data integration, advanced analytics, and modern applications that address new customer needs and are highly visible and strategic within the organization. Do you...

  • Senior Engineer, Data Engineering

    4 days ago

    India Vista Applied Solutions Group Inc Full time

    Job Summary: Our client is seeking a Senior Data Engineer to lead the modernization of our enterprise data platform using Microsoft Fabric, with a focus on building an AI-ready foundation. This role is key to replacing legacy ERP systems, retiring SSIS, and implementing scalable, code-driven data pipelines to support advanced analytics and future AI...

  • Pipelines Technician

    2 weeks ago

    India Mott MacDonald Full time

    **Key Responsibilities** Develop project concepts, prepare construction documents, and support design tasks on assigned projects that include: - Support alternative analysis (route/alignment, pipe size and material, pump sizing, etc.) for use in business case evaluations and decision making. - Develop pipeline alignments and Pipeline Route map drawings -...

  • Senior data engineer

    5 days ago

    India Vista Applied Solutions Group Inc Full time

    Job Summary: Our client is seeking a Senior Data Engineer to lead the modernization of our enterprise data platform using Microsoft Fabric, with a focus on building an AI-ready foundation. This role is key to replacing legacy ERP systems, retiring SSIS, and implementing scalable, code-driven data pipelines to support advanced analytics and future AI...

  • Senior Data Engineer

    5 days ago

    India Vista Applied Solutions Group Inc Full time

    Job Summary: Our client is seeking a Senior Data Engineer to lead the modernization of our enterprise data platform using Microsoft Fabric, with a focus on building an AI-ready foundation. This role is key to replacing legacy ERP systems, retiring SSIS, and implementing scalable, code-driven data pipelines to support advanced analytics and future AI...

  • Senior Data Pipeline Developer

    2 weeks ago

    India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    We are seeking a skilled ETL Developer / Senior Developer to join our data engineering team in a full-time capacity.The successful candidate will work on building scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.Key ResponsibilitiesDesign, develop, and maintain ETL pipelines using IBM DataStage (CP4D)...

  • Backend and Data Pipeline Engineer

    1 day ago

    India JRD Systems Full time

    Job Role: Backend and Data Pipeline Engineer - Python Location: Remote Job Type : Fulltime ** Only Immediate Joiners ** Job Summary: The Team: We’re investing in technology to develop new products that help our customers drive their growth and transformation agenda. These include new data integration, advanced analytics, and modern applications that...

  • Backend and Data Pipeline Engineer

    8 hours ago

    India JRD Systems Full time

    Job Role: Backend and Data Pipeline Engineer - PythonLocation: RemoteJob Type : Fulltime** Only Immediate Joiners **Job Summary:The Team:We’re investing in technology to develop new products that help our customers drive their growth and transformation agenda. These include new data integration, advanced analytics, and modern applications that address new...