Data Pipeline Architect

2 weeks ago

Kanpur, Uttar Pradesh, India beBeeData Full time US$ 1,11,111 - US$ 1,53,333
**Job Title:** Lead Data Platform Engineer**Job Description:The role of the Lead Data Platform Engineer is to oversee the design, development, and operation of data pipelines. As a technical leader, you will be responsible for defining technical direction, running design reviews, coaching engineers, and coordinating work across time zones.Key Responsibilities include:
  • Technical leadership & delivery: Manage a small squad or workstream; scope milestones; manage dependencies and risks; unblock engineers; communicate status to stakeholders; land outcomes on time.
  • Architecture & direction: Define target states and data contracts; make build/buy calls; author RFCs; drive cost/performance guardrails and reliability patterns.
  • Data pipelines: Design, build, and operate reliable batch and change-driven pipelines; schedule/orchestrate jobs; handle schema evolution and failures gracefully.
  • Transformations & modeling: Implement clean, tested merge/transformation pipelines; produce models that are easy to consume for products and analytics; tune SQL for performance.
  • APIs & integration: Build or collaborate on APIs for data access/updates; design request/response contracts; handle auth (OAuth2/OIDC/JWT), idempotency, validation, and auditing.
  • Infrastructure & DevOps: Provision/manage cloud resources with Terraform; automate build/test/deploy with CI/CD; write solid shell scripts for glue/ops tasks.
  • Observability & reliability: Instrument logs/metrics/traces; define SLOs and on-call runbooks; lead incident reviews and drive root-cause prevention.
  • Security & compliance: Apply least-privilege access, secrets management, and encryption in transit/at rest; partner on compliance requirements.
  • Quality & standards: Champion unit/integration/data-quality tests (including SQL); enforce code health via reviews, linters, and CI checks.
  • Collaboration & docs: Operate async-first; produce concise docs (data contracts, mappings, runbooks, RFCs); collaborate closely with product/engineering/analytics across time zones.
**Required Skills & Qualifications:**
  • Proven leadership: led engineers or owned large, multi-team projects; comfortable running design reviews and giving actionable feedback.
  • Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL).
  • Python for data work and services; comfortable with shell scripting (bash preferred; Powershell acceptable).
  • Terraform and IaC fundamentals; CI/CD experience (version control workflows, automated tests, environment promotion).
  • Cloud: Azure preferred (storage, compute, identity, monitoring). If not Azure, very strong AWS background.
  • Data integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays.
  • Workflow orchestration: Prefect preferred (flows/deployments, retries/backoff, scheduling, observability); Airflow/Dagster also acceptable.
  • APIs & auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.
  • Observability: Logs/metrics/traces and actionable alerts/dashboards.
  • Testing mindset: Habit of writing tests for code and SQL; fixtures/test data; CI.
  • Communication: Clear written/verbal comms; practiced cross-timezone collaboration.
  • AI tooling: Ready to use Claude Code (and similar assistants) to accelerate work—while exercising judgment and writing your own code/tests.
**Benefits:**
  • Duration: 10+ months (extension/full-time possible).
  • Schedule: Flexible, with overlap for key meetings across time zones.
  • Compensation: Competitive, based on experience.
  • Mode: Remote; available for virtual collaboration and on-call windows as agreed.

  • Data Pipeline Architect

    2 weeks ago

    Kanpur, Uttar Pradesh, India beBeeDataPipeline Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Pipeline ArchitectWe are seeking an experienced professional to design and build scalable data pipelines for processing channel activity data from multiple sources.The ideal candidate will possess strong knowledge of AWS cloud infrastructure, particularly services like Pyspark, Glue, EMR, lambda, Redshift, Athena, and other tools like...

  • Data Pipeline Architect

    1 week ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time US$ 10,00,000 - US$ 15,00,000

    Unlocking Data PotentialA unified commerce platform is revolutionizing the food and beverage industry with its cutting-edge software solutions. The company empowers businesses across multiple countries to thrive in today's fast-paced market.A team of developers, designers, and product experts drives innovation, introducing features that set industry...

  • Advanced Data Pipeline Architect

    2 weeks ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Data Systems DeveloperWe are seeking a skilled Data Systems Developer to join our team. As a key member of our data engineering team, you will be responsible for designing and implementing efficient data pipelines that drive business insights and decision-making.This is a mid-level role ideal for someone with 3-5 years of experience in data...

  • CV Pipeline Architect

    1 week ago

    Kanpur, Uttar Pradesh, India beBeeComputerVision Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Computer Vision Expert WantedWe are seeking a skilled Computer Vision Expert to lead the design and implementation of complex pipelines. The ideal candidate will have expertise in multi-object tracking, object detection, and semi/unsupervised learning.Key Responsibilities:Design and implement complex CV pipelines (object detection, instance segmentation,...

  • Data Pipeline Specialist

    2 weeks ago

    Kanpur, Uttar Pradesh, India beBeeBusiness Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Business Intelligence Developer","Role Summary:","We are seeking an experienced professional to design and implement scalable data pipelines using industry-standard tools.","Key Responsibilities:","Design and implement scalable data pipelines.","Develop ETL/ELT solutions within a cloud-based environment.","Manage and optimize data storage and...

  • Cloud Data Architect

    2 weeks ago

    Kanpur, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Data ArchitectWe are seeking a Cloud Data Architect to design and develop scalable data pipelines and ETL solutions in a cloud environment using AWS services. This individual will collaborate with cross-functional teams to support data-driven initiatives. Key Responsibilities: Build and maintain complex data architectures on AWS, utilizing its suite...

  • Data Architect

    1 week ago

    Kanpur, Uttar Pradesh, India beBeeDatabase Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Job Title: Data ArchitectThe ideal candidate will have a strong command of SQL, experience with ETL processes, and a deep understanding of data modelling and database performance tuning.About the Role:Design, develop, and maintain complex relational databases.Write efficient and optimized SQL queries, stored procedures, functions, views, and triggers.Develop...

  • Data Pipeline Specialist

    2 weeks ago

    Kanpur, Uttar Pradesh, India beBeeMachine Full time ₹ 25,00,000 - ₹ 28,00,000

    Senior Machine Learning EngineerKey Responsibilities:Develop and optimize complex data pipelines, applying machine learning engineering principles to enhance efficiency and scalability.Integrate and optimize data and model pipelines within production environments, diagnosing data inconsistencies and documenting assumptions.Employ experimental methodologies,...

  • Cloud Data Architect

    1 week ago

    Kanpur, Uttar Pradesh, India beBeeData Full time US$ 12,00,000 - US$ 20,00,000

    Job Title: Cloud Data ArchitectWe are seeking an experienced professional to design, build, and optimize scalable data pipelines and architectures on Google Cloud Platform.Requirements include 5+ years of experience in data engineering with a proven track record of designing and optimizing data systems at scale.The ideal candidate is skilled in managing...

  • Data Architect

    5 days ago

    Kanpur, Uttar Pradesh, India beBeeData Full time ₹ 6,00,000 - ₹ 12,00,000

    Job Title:"Business Intelligence Developer"Job Description:We are seeking an experienced Business Intelligence developer to join our team. The ideal candidate will have strong technical skills and a passion for delivering high-quality solutions.Key Responsibilities:Design, develop and maintain robust data integration pipelines using Informatica IDMC...