Lead Data Platform Engineer

Position: Lead Data Platform Engineer

Location: Remote (collaboration across time zones), India or LATAM preferred

Engagement Type: Contract

Language: Fluent English

Experience: 9+ years in Data Engineering / Backend Engineering / DevOps, including leadership of a team (or end-to-end ownership of large, multi-stakeholder projects)

About the Role

We're looking for a hands-on Lead Data Platform Engineer to both build/operate production-grade data pipelines and lead delivery. You'll set technical direction, run design reviews and code reviews, coach engineers, and coordinate work across time zones. Day to day you'll work across ingestion, transformation, orchestration, and APIs; automate environments with Terraform; instrument observability; and write tests to keep things reliable. Azure experience is preferred; strong AWS experience is also acceptable. You should communicate clearly (async chat and live sessions), learn fast, and use AI coding tools (e.g., Claude Code) effectively without relying on them exclusively.

Key Responsibilities

• Technical leadership & delivery: Lead a small squad or workstream; scope milestones; manage dependencies and risks; unblock engineers; communicate status to stakeholders; land outcomes on time
• Architecture & direction: Define target states and data contracts; make build/buy calls; author RFCs; drive cost/performance guardrails and reliability patterns
• Data pipelines: Design, build, and operate reliable batch and change-driven pipelines; schedule/orchestrate jobs; handle schema evolution and failures gracefully
• Transformations & modeling: Implement clean, tested merge/transformation pipelines; produce models that are easy to consume for products and analytics; tune SQL for performance
• APIs & integration: Build or collaborate on APIs for data access/updates; design request/response contracts; handle auth (OAuth2/OIDC/JWT), idempotency, validation, and auditing
• Infrastructure & DevOps: Provision/manage cloud resources with Terraform; automate build/test/deploy with CI/CD; write solid shell scripts for glue/ops tasks
• Observability & reliability: Instrument logs/metrics/traces; define SLOs and on-call runbooks; lead incident reviews and drive root-cause prevention
• Security & compliance: Apply least-privilege access, secrets management, and encryption in transit/at rest; partner on compliance requirements
• Quality & standards: Champion unit/integration/data-quality tests (including SQL); enforce code health via reviews, linters, and CI checks
• Collaboration & docs: Operate async-first; produce concise docs (data contracts, mappings, runbooks, RFCs); collaborate closely with product/engineering/analytics across time zones

Required Skills & Qualifications

• Proven leadership: led engineers or owned large, multi-team projects; comfortable running design reviews and giving actionable feedback
• Strong SQL (tuning, indexing, query plans) and solid RDBMS fundamentals (e.g., Postgres, SQL Server, MySQL)
• Python for data work and services; comfortable with shell scripting (bash preferred; Powershell acceptable)
• Terraform and IaC fundamentals; CI/CD experience (version control workflows, automated tests, environment promotion)
• Cloud: Azure preferred (storage, compute, identity, monitoring). If not Azure, very strong AWS background
• Data integration: Ingestion (batch + change-driven), orchestration, schema/versioning, resilient retries/replays
• Workflow orchestration: Prefect preferred (flows/deployments, retries/backoff, scheduling, observability); Airflow/Dagster also acceptable
• APIs & auth: REST patterns, pagination, validation, rate limiting, OAuth2/OIDC/JWT.
• Observability: Logs/metrics/traces and actionable alerts/dashboards
• Testing mindset: Habit of writing tests for code and SQL; fixtures/test data; CI
• Communication: Clear written/verbal comms; practiced cross-timezone collaboration
• AI tooling: Ready to use Claude Code (and similar assistants) to accelerate work—while exercising judgment and writing your own code/tests

Preferred Qualifications (Nice to Have)

• Experience mentoring, interviewing, onboarding, and raising code quality standards
• Regulatory/compliance awareness (e.g., data protection or public-sector standards) and how it impacts design/operations
• Analytics exposure (dim/fact modeling, BI consumption patterns)
• Data platform architecture exposure (storage/layout choices, catalog/lineage, governance concepts)
• Familiarity with Kafka, streaming/message queues, and columnar formats (e.g., Parquet)
• Experience with cloud monitoring stacks (Azure Monitor/Application Insights, CloudWatch, Datadog, New Relic, etc.)

Engagement Details

• Duration: 10+ months (extension/full-time possible)
• Schedule: Flexible, with overlap for key meetings across time zones
• Compensation: Competitive, based on experience
• Mode: Remote; available for virtual collaboration and on-call windows as agreed

How to Apply

Please share:

• Resume/CV highlighting data engineering, DevOps/IaC, and/or API/auth work
• Brief note (one paragraph) about a recent pipeline/service: requirements, your role, tech, validation/tests, deployment/ops, outcome
• Links to code samples or design docs (if available/shareable)

Apply via or email careers@blackdoglabs.io.