Senior Reliability Engineer

Senior Reliability Engineer

We are seeking an experienced Senior Reliability Engineer to join our team. In this role, you will be responsible for ensuring the high availability, performance, and efficiency of our SaaS application on Azure. This includes defining and enforcing reliability standards, leading high-impact projects, mentoring engineers, and eliminating toil at scale.

Key Responsibilities:

• Reliability Standards: Define customer-centric SLIs/SLOs for Tier-0/Tier-1 services and publish quarterly reviews. Align teams with these standards to ensure everyone is working towards the same goals.
• Error Budgeting: Implement error-budget policy with multi-window, multi-burn-rate alerts; clear runbooks and paging thresholds. Gate changes by budget status (freeze/relax rules) wired into CI/CD.
• SLO/EB Dashboards: Maintain SLO/EB dashboards using Azure Monitor, Grafana/Prometheus, and App Insights. Run weekly SLO reviews with engineering/product teams.
• Incident Response: Lead SEV1/SEV2 incidents without drama. Own comms, run blameless postmortems, and make corrective actions stick.
• Engineering Reliability: Engineer reliability in multi-AZ/region patterns (active-active/DR), PDBs/Pod Topology Spread, HPA/VPA/KEDA, resilient rollout/rollback.
• Azure Expertise: Harden AKS clusters (network, identity, policy), optimize node/pod density, ingress (AGIC/Nginx); mesh optional.
• Observability: Metrics/traces/logs with Azure Monitor/App Insights, Log Analytics, Prometheus/Grafana, OpenTelemetry. Alert on symptoms, not noise.
• IaC & Policy: Terraform/Bicep modules, GitOps (Flux/Argo), policy-as-code (Azure Policy/OPA Gatekeeper). No snowflakes.
• CI/CD Reliability: Azure DevOps/GitHub Actions with canary/blue-green, progressive delivery, auto-rollback, Key Vault-backed secrets.
• Capacity & Performance: Load testing, right-sizing, autoscaling; partner with FinOps to reduce spend without hurting SLOs.
• Disaster Recovery: Define RTO/RPO, test backups/restore, run game days/chaos drills, validate ASR and multi-region failover.
• Security: Entra ID (Azure AD), managed identities, Key Vault rotation, VNets/NSGs/Private Link, shift-left checks in CI.
• Toil Reduction: Automate recurring ops, build self-service runbooks/chatops, publish golden paths for product teams.
• Customer Escalations: Be the technical owner on calls; communicate tradeoffs and recovery plans with authority.
• Documentation: Architectures, runbooks, postmortems, SLIs/SLOs—kept current and discoverable.
• Data Reliability: Apply SRE practices (SLOs, backpressure, idempotency, replay) to NiFi/Flink/Kafka/Redpanda data flows.

Requirements:

• Bachelor's degree in Computer Science or related field.
• At least 12 years of experience in production operations, platform engineering, or SRE, including 5+ years on Azure.
• Deep operational expertise in PostgreSQL, including HA/DR, logical/physical replication, performance tuning, autovacuum strategy, partitioning, backup/restore testing, and connection pooling.
• Azure core skills: AKS, Front Door/App Gateway, API Management, VNets/NSGs/Private Link, Storage, Key Vault, Redis, Service Bus/Event Hubs.
• Observability skills: Azure Monitor/App Insights, Log Analytics, Prometheus/Grafana, SLO design, error-budget operations.
• IaC/automation skills: Terraform and/or Bicep, PowerShell and Python, GitOps (Flux/Argo). Pipelines in Azure DevOps or GitHub Actions.
• Proven incident leadership at scale, blameless postmortems, SLO/error-budget governance with change gating.
• Mentorship and crisp written/verbal communication.

PREFERRED SKILLS:

• Apache NiFi, Apache Flink, Apache Kafka or Redpanda; schema management, exactly-once semantics, backpressure, dead-letter/replay patterns.
• Azure Solutions Architect Expert, CKA/CKAD.
• ITSM (ServiceNow), on-call tooling (PagerDuty/Opsgenie).
• Compliance/SecOps (SOC 2, ISO 27001), policy-as-code, workload identity.
• OpenTelemetry, eBPF tooling, or service mesh.
• Multi-tenant SaaS and cost optimization at scale.