Technical Lead for Data Pipelines

2 days ago

Bareilly, Uttar Pradesh, India beBeeDataPlatformEngineer Full time US$ 1,25,000 - US$ 1,75,000
Job Overview

We are seeking a seasoned Technical Leader to spearhead the development and operation of production-grade data pipelines. As a Lead Data Platform Engineer, you will be responsible for setting technical direction, running design reviews, coaching engineers, and coordinating work across time zones.

Key Responsibilities:
  • Data Pipeline Engineering: Design, build, and operate reliable batch and change-driven pipelines, schedule/orchestrate jobs, handle schema evolution and failures gracefully.
  • Architecture & Direction: Define target states and data contracts, make build/buy calls, author RFCs, drive cost/performance guardrails and reliability patterns.
  • Data Integration: Implement clean, tested merge/transformation pipelines, produce models that are easy to consume for products and analytics, tune SQL for performance.
  • API Development: Build or collaborate on APIs for data access/updates, design request/response contracts, handle auth (OAuth2/OIDC/JWT), idempotency, validation, and auditing.
  • Infrastructure Management: Provision/manage cloud resources with Terraform, automate build/test/deploy with CI/CD, write solid shell scripts for glue/ops tasks.
  • Observability & Reliability: Instrument logs/metrics/traces, define SLOs and on-call runbooks, lead incident reviews and drive root-cause prevention.
  • Security & Compliance: Apply least-privilege access, secrets management, and encryption in transit/at rest, partner on compliance requirements.
  • Quality Assurance: Champion unit/integration/data-quality tests (including SQL), enforce code health via reviews, linters, and CI checks.
  • Collaboration & Documentation: Operate async-first, produce concise docs (data contracts, mappings, runbooks, RFCs), collaborate closely with product/engineering/analytics across time zones.
Required Skills & Qualifications:
  • Proven leadership: led engineers or owned large, multi-team projects; comfortable running design reviews and giving actionable feedback.
  • Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL).
  • Python for data work and services; comfortable with shell scripting (bash preferred; Powershell acceptable).
  • Terraform and IaC fundamentals; CI/CD experience (version control workflows, automated tests, environment promotion).
  • Cloud: Azure preferred (storage, compute, identity, monitoring). If not Azure, very strong AWS background.
  • Data integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays.
  • Workflow orchestration: Prefect preferred (flows/deployments, retries/backoff, scheduling, observability); Airflow/Dagster also acceptable.
  • APIs & auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.
  • Observability: Logs/metrics/traces and actionable alerts/dashboards.
  • Testing mindset: Habit of writing tests for code and SQL; fixtures/test data; CI.
  • Communication: Clear written/verbal comms; practiced cross-timezone collaboration.
  • AI tooling: Ready to use Claude Code (and similar assistants) to accelerate work—while exercising judgment and writing your own code/tests.
Benefits:
  • Flexible schedule with overlap for key meetings across time zones.
  • Competitive compensation based on experience.
  • Remote work option with virtual collaboration and on-call windows as agreed.

Please share your resume/CV highlighting data engineering, DevOps/IaC, and/or API/auth work, a brief note about a recent pipeline/service, and links to code samples or design docs (if available/shareable).

],

  • Lead Data Pipeline Specialist

    1 week ago

    Bareilly, Uttar Pradesh, India beBeeTester Full time ₹ 15,00,000 - ₹ 25,00,000

    About this position :We are seeking a highly skilled ETL and Power BI Tester to join our team. As an experienced professional, you will be responsible for ensuring the accuracy, integrity, and performance of ETL Pipelines deployed into Azure cloud environments.The role involves developing and automating comprehensive test cases to validate data flows,...

  • Senior Python Data Pipeline Specialist

    1 week ago

    Bareilly, Uttar Pradesh, India beBeeEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title:Freelance Data Engineer – Python\Data Engineering Expertise RequiredThreatXIntel is a cyber security company that protects businesses and organizations from cyber threats.We offer customized and affordable solutions tailored to meet the specific needs of our clients, ensuring that businesses of all sizes have access to high-quality cyber security...

  • Chief Data Pipeline Developer

    2 days ago

    Bareilly, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Engineering ProfessionalWe are seeking a highly skilled professional to lead the development and maintenance of robust data pipelines for structured and unstructured data.Key Responsibilities:Hold a bachelor's or master's degree in Computer Science, Data Science, Engineering, or a related field with 3+ years of experience.Proven expertise in working...

  • Data Pipeline Engineer

    3 days ago

    Bareilly, Uttar Pradesh, India beBeeDataintegration Full time ₹ 15,00,000 - ₹ 30,00,000

    ETL Developer Job DescriptionWe are seeking an experienced ETL developer to design and develop complex data processing systems.The ideal candidate will have hands-on experience with data integration tools and a strong understanding of SQL, UNIX, and scripting languages.Prior experience with cloud-based data platforms is also highly valued.This role involves...

  • Advanced Data Pipeline Engineer

    6 days ago

    Bareilly, Uttar Pradesh, India beBeeBlockchain Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Backend Developer Role SummaryThe role involves building robust data pipelines that power real-time pricing engines, decoding smart contract events, and implementing pricing logic from decentralized exchanges (DEXs).Job Description:We are seeking a skilled Backend Developer to join our team. The ideal candidate will have experience in Go or Rust, a strong...

  • Technical Lead in Data Architecture

    15 hours ago

    Bareilly, Uttar Pradesh, India beBeeDataArchitecture Full time ₹ 30,00,000 - ₹ 45,00,000

    Job Title: Technical Lead in Data ArchitectureJob Description:We are seeking a seasoned technical lead to drive the design and implementation of our data architecture strategy.The successful candidate will have a strong background in data engineering, experience with cloud-based technologies, and excellent communication skills.This is an exciting opportunity...

  • Lead Data Architect

    2 days ago

    Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Data Architect PositionWe are seeking an experienced Data Architect to lead the design and implementation of our data platforms.Key Responsibilities:Design, develop and deploy scalable data architectures that meet business requirements.Collaborate with cross-functional teams to identify and implement data solutions.Develop technical strategies for data...

  • Lead Data Strategist

    2 weeks ago

    Bareilly, Uttar Pradesh, India beBeeDataSpecialist Full time US$ 1,75,000 - US$ 2,25,000

    Data SpecialistWe are seeking a skilled Data Specialist to lead the modernization of our clients' data infrastructures, architecture, and pipelines.Key Responsibilities:Develop cloud-first strategies for core business data initiativesProvide thought leadership on technical mattersDesign and implement modernized end-to-end data strategies, including...

  • High-Performance Data Pipeline Specialist

    2 weeks ago

    Bareilly, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Databricks Data Engineer Job DescriptionWe are seeking a highly skilled Senior Databricks Data Engineer to design, develop, and maintain scalable and high-performance data pipelines using Databricks and PySpark.The ideal candidate will have strong hands-on experience in Databricks and AWS, with a solid understanding of PySpark, CI/CD pipelines, and...

  • Technical Lead

    2 days ago

    Bareilly, Uttar Pradesh, India beBeeExperience Full time ₹ 2,00,00,000 - ₹ 3,50,00,000

    Job Opportunity: Technical LeaderWe are seeking a highly skilled technical leader to join our core engineering team. As a hands-on architect, you will be responsible for delivering and owning business-critical platforms end-to-end.Responsibilities:Own architecture & delivery of payment, catalog, logistics, and growth services handling tens of thousands of...