Chief Data Pipeline Architect

Job Description:

This role focuses on building and operating production-grade data pipelines. You will lead delivery, set technical direction, run design reviews, coach engineers, and coordinate work across time zones.

Day-to-day tasks include working across ingestion, transformation, orchestration, and APIs; automating environments with Terraform; instrumenting observability; and writing tests to keep things reliable.

Azure experience is preferred; strong AWS experience is also acceptable.

Key responsibilities include:

• ">
• Technical leadership & delivery: Lead a small squad or workstream; scope milestones; manage dependencies and risks; unblock engineers; communicate status to stakeholders; land outcomes on time">
• Architecture & direction: Define target states and data contracts; make build/buy calls; author RFCs; drive cost/performance guardrails and reliability patterns">
• Data pipelines: Design, build, and operate reliable batch and change-driven pipelines; schedule/orchestrate jobs; handle schema evolution and failures gracefully">
• Transformations & modeling: Implement clean, tested merge/transformation pipelines; produce models that are easy to consume for products and analytics; tune SQL for performance">
• APIs & integration: Build or collaborate on APIs for data access/updates; design request/response contracts; handle auth (OAuth2/OIDC/JWT), idempotency, validation, and auditing">
• Infrastructure & DevOps: Provision/manage cloud resources with Terraform; automate build/test/deploy with CI/CD; write solid shell scripts for glue/ops tasks">
• Observability & reliability: Instrument logs/metrics/traces; define SLOs and on-call runbooks; lead incident reviews and drive root-cause prevention">
• Security & compliance: Apply least-privilege access, secrets management, and encryption in transit/at rest; partner on compliance requirements">
• Quality & standards: Champion unit/integration/data-quality tests (including SQL); enforce code health via reviews, linters, and CI checks">
• Collaboration & docs: Operate async-first; produce concise docs (data contracts, mappings, runbooks, RFCs); collaborate closely with product/engineering/analytics across time zones">

Required Skills & Qualifications:

• ">
• Proven leadership: led engineers or owned large, multi-team projects; comfortable running design reviews and giving actionable feedback">
• Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL)">
• Python for data work and services; comfortable with shell scripting (bash preferred; Powershell acceptable)">
• Terraform and IaC fundamentals; CI/CD experience (version control workflows, automated tests, environment promotion)">
• Cloud: Azure preferred (storage, compute, identity, monitoring). If not Azure, very strong AWS background">
• Data integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays">
• Workflow orchestration: Prefect preferred (flows/deployments, retries/backoff, scheduling, observability); Airflow/Dagster also acceptable">
• APIs & auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.">
• Observability: Logs/metrics/traces and actionable alerts/dashboards">
• Testing mindset: Habit of writing tests for code and SQL; fixtures/test data; CI">
• Communication: Clear written/verbal comms; practiced cross-timezone collaboration">
• AI tooling: Ready to use AI coding tools to accelerate work—while exercising judgment and writing your own code/tests">

Preferred Qualifications (Nice to Have):

• ">
• Experience mentoring, interviewing, onboarding, and raising code quality standards">
• Regulatory/compliance awareness (e.g., data protection or public-sector standards) and how it impacts design/operations">
• Analytics exposure (dim/fact modeling, BI consumption patterns)">
• Data platform architecture exposure (storage/layout choices, catalog/lineage, governance concepts)">
• Familiarity with Kafka, streaming/message queues, and columnar formats (e.g., Parquet)">
• Experience with cloud monitoring stacks (Azure Monitor/Application Insights, CloudWatch, Datadog, New Relic, etc.)">

Engagement Details:

• ">
• Duration: 10+ months (extension/full-time possible)">
• Schedule: Flexible, with overlap for key meetings across time zones">
• Compensation: Competitive, based on experience">
• Mode: Remote; available for virtual collaboration and on-call windows as agreed">

How to Apply:

Please share:

• ">
• Resume/CV highlighting data engineering, DevOps/IaC, and/or API/auth work">
• Brief note (one paragraph) about a recent pipeline/service: requirements, your role, tech, validation/tests, deployment/ops, outcome">
• Links to code samples or design docs (if available/shareable)">