Chief Data Pipeline Architect

We are seeking an experienced Lead Data Platform Engineer to join our organization. As a key member of the engineering team, you will be responsible for designing and building production-grade data pipelines, as well as leading delivery and technical direction.

The ideal candidate will have 9+ years of experience in Data Engineering/Backend Engineering/DevOps, including leadership of a team or end-to-end ownership of large, multi-stakeholder projects. You will need to communicate effectively, learn quickly, and use AI coding tools judiciously without relying on them exclusively.

• Key Responsibilities:
• Technical Leadership & Delivery: Scope milestones, manage dependencies and risks, unblock engineers, communicate status to stakeholders, achieve outcomes on time.
• Architecture & Direction: Define target states and data contracts, make strategic build/buy calls, author Request for Comments (RFCs), drive cost/performance guardrails and reliability patterns.
• Data Pipelines: Design, build, and operate reliable batch and change-driven pipelines, schedule/orchestrate jobs, handle schema evolution and failures with resilience.
• Transformations & Modeling: Implement clean, tested merge/transformation pipelines, produce models that are easy to consume for products and analytics, tune SQL for optimal performance.
• APIs & Integration: Build or collaborate on APIs for data access/updates, design request/response contracts, handle authentication (OAuth2/OIDC/JWT), idempotency, validation, and auditing.
• Infrastructure & DevOps: Provision/manage cloud resources using Terraform, automate build/test/deploy with Continuous Integration/Continuous Deployment (CI/CD), write robust shell scripts for glue/ops tasks.
• Observability & Reliability: Instrument logs/metrics/traces, define Service Level Objectives (SLOs) and on-call runbooks, lead incident reviews and drive root-cause prevention.
• Security & Compliance: Apply least-privilege access, secrets management, and encryption in transit/at rest, partner on compliance requirements.
• Quality & Standards: Champion unit/integration/data-quality tests (including SQL), enforce code health via reviews, linters, and CI checks.
• Collaboration & Docs: Operate asynchronously, produce concise documentation (data contracts, mappings, runbooks, RFCs), collaborate closely with product/engineering/analytics across time zones.

• Proven Leadership: Led engineers or owned large, multi-team projects; comfortable running design reviews and giving actionable feedback.
• Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL).
• Python for data work and services; comfortable with shell scripting (bash preferred; Powershell acceptable).
• Terraform and Infrastructure as Code (IaC) fundamentals; CI/CD experience (version control workflows, automated tests, environment promotion).
• Cloud: Azure preferred (storage, compute, identity, monitoring). If not Azure, very strong AWS background.
• Data Integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays.
• Workflow Orchestration: Prefect preferred (flows/deployments, retries/backoff, scheduling, observability); Airflow/Dagster also acceptable.
• APIs & Auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.
• Observability: Logs/metrics/traces and actionable alerts/dashboards.
• Testing Mindset: Habit of writing tests for code and SQL; fixtures/test data; CI.
• Communication: Clear written/verbal comms; practiced cross-timezone collaboration.
• Ai Tooling: Ready to use Claude Code (and similar assistants) to accelerate work—while exercising judgment and writing your own code/tests.