Senior Pipeline Engineer

3 days ago

India Weekday AI Full time ₹ 15,00,000 - ₹ 20,00,000 per year

This role is for one of Weekday's clients
Min Experience: 6 years
Location: India
JobType: full-time

We are looking for an experienced and motivated Senior Pipeline Engineer to join our engineering team and take ownership of designing, building, and managing high-performance data pipelines. This role demands strong expertise in orchestration frameworks such as Celery and Temporal, combined with advanced skills in database design and query optimization. The ideal candidate will have a passion for solving complex technical challenges and delivering reliable, scalable solutions that power data-driven decision-making across the organization.

As a senior team member, you will play a critical role in shaping the future of our data infrastructure, working closely with cross-functional teams including data scientists, backend engineers, and product managers. You will also mentor junior engineers and drive best practices in orchestration, database efficiency, and scalable pipeline design.

Key Responsibilities
  • Pipeline Development & Management
    • Design, develop, and maintain robust and scalable data pipelines for batch and real-time processing.
    • Automate workflows for data ingestion, transformation, validation, and delivery.
    • Ensure pipelines are fault-tolerant, performant, and cost-efficient.
  • Workflow Orchestration
    • Build and manage orchestration workflows using Celery, Temporal, or similar tools.
    • Monitor, schedule, and optimize pipeline execution to reduce latency and improve reliability.
    • Implement scalable solutions for dependency management, retries, and error handling.
  • Database Design & Query Optimization
    • Create efficient database schemas and models to support large-scale data operations.
    • Optimize complex queries, indexing, and partitioning strategies to ensure high performance.
    • Collaborate with engineering teams to enforce database best practices and scalability standards.
  • Collaboration & Leadership
    • Partner with product managers, analysts, and business stakeholders to understand requirements and translate them into technical solutions.
    • Mentor and guide junior engineers in pipeline development and orchestration practices.
    • Contribute to architecture reviews, design discussions, and technical documentation.
  • Continuous Improvement & Compliance
    • Proactively identify opportunities to improve pipeline performance and reliability.
    • Stay updated on industry trends, orchestration frameworks, and database technologies.
    • Ensure adherence to data security, governance, and compliance policies.
Required Skills & Qualifications
  • 6+ years of experience in pipeline engineering, data engineering, or related fields.
  • Strong hands-on expertise in Celery, Temporal, or other orchestration frameworks.
  • Proficiency in database design, query optimization, and performance tuning.
  • Solid programming background in Python, Java, or similar languages.
  • Deep understanding of distributed systems, messaging queues, and event-driven architectures.
  • Experience with cloud platforms (AWS, Azure, GCP) and containerized environments (Docker, Kubernetes).
  • Strong problem-solving, analytical, and debugging skills.
  • Excellent communication and collaboration abilities.
Preferred Skills
  • Familiarity with real-time data streaming platforms like Kafka or Pulsar.
  • Experience with data warehousing tools (Snowflake, BigQuery, Redshift).
  • Knowledge of CI/CD pipelines, DevOps practices, and infrastructure automation.

  • Senior Engineer

    4 weeks ago

    India Xebia Full time

    Senior Engineer (Gen AI & Prompt Engineering) | Xebia | Hybrid (India) We're hiring a Gen AI expert to supercharge our Dev Ops and Engineering ecosystem with AI-native intelligence.If you're passionate about Prompt Engineering, RAG pipelines, and vector databases , and can deliver production-ready LLM integrations — we want to hear from you Key...

  • Senior Data Engineer

    3 weeks ago

    India Asenium Consulting Full time

    For one of my customers, we are looking for a Senior Data Engineer.Location: India (100% remote)Duration: 6+ months (renewable)Description: We are looking for a Senior Data Engineer with strong experience in SQL, Talend, and cloud platforms such as Google Cloud (Big Query) and Microsoft Azure.You will be responsible for designing and managing ETL pipelines,...

  • Senior Azure Databricks Engineer

    4 weeks ago

    India Quant-data Full time

    We're Hiring: Senior Azure Databricks Engineer (Remote | Full-Time) Join us to build scalable, cloud-native data solutions using cutting-edge technologiesWe're looking for an experienced and driven Senior Azure Databricks Engineer to join our growing team. This role is ideal for professionals with 6–10 years of experience in the data engineering space,...

  • Senior Data Engineer

    4 weeks ago

    India Asenium Consulting Full time

    For one of my customers, we are looking for a Senior Data Engineer.Location: India (100% remote)Duration: 6+ months (renewable)Description:We are looking for a Senior Data Engineer with strong experience in SQL, Talend, and cloud platforms such as Google Cloud (BigQuery) and Microsoft Azure. You will be responsible for designing and managing ETL pipelines,...

  • Senior Engineer

    2 weeks ago

    India Xebia Full time

    Senior Engineer (GenAI & Prompt Engineering) | Xebia We are looking for a highly experienced GenAI Engineer with deep expertise in Prompt Engineering, Retrieval-Augmented Generation (RAG), and Vector Search Systems to integrate GenAI into our engineering & DevOps ecosystem. This is a contractor role where you will design intelligent assistants &...

  • Senior Quality Engineer

    4 weeks ago

    India IntraEdge Full time

    Job Title: Senior QA Automation Engineer – API & CypressExperience Level: Senior (5–10 years) Location: [Remote] Employment Type: Full-timeAbout the Role:We are looking for a Senior QA Automation Engineer with strong hands-on experience in API and UI test automation using Cypress. In this role, you will lead test strategy, design robust automation...

  • Senior Sre Engineer

    4 weeks ago

    India BugRaid AI Full time

    Company Description Bug Raid.AI adopts advanced AIOPS and AI bots for proactive incident management and response, transforming the entirety of the process.By integrating sophisticated AIOPS for comprehensive incident analysis with AI bots for immediate response, Bug Raid.AI provides automated and intelligent incident handling.Our platform enables...

  • Senior Data Engineer

    2 weeks ago

    India 3Pillar Global Full time

    Job DescriptionEmbark on an exciting journey into the realm of software development with 3Pillar We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making.As a Senior Data Engineer, you will hold...

  • Senior Performance Engineer- Hybrid Working

    2 weeks ago

    India Xebia Full time

    Senior Engineer (GenAI & Prompt Engineering) | Xebia We are looking for a highly experienced GenAI Engineer with deep expertise in Prompt Engineering, Retrieval-Augmented Generation (RAG), and Vector Search Systems to integrate GenAI into our engineering & DevOps ecosystem. This is a contractor role where you will design intelligent assistants & agents ...

  • Senior Data Engineer

    3 weeks ago

    India Aptus Data Labs Full time

    Job Title: Senior Data EngineerLocation: RemoteExperience: 4+Employment Type: Full-TimeNotice- Looking for immediate joiners or who can join within 15-20 days.About the RoleAptus Data Labs is looking for a talented and proactive Senior Data Engineer to help build the backbone of our enterprise data and AI initiatives. You'll work on modern data lake...