Senior Python Platform Engineer

Transform Your Career with Our Senior Python Platform Role

Are you a seasoned Python engineer looking to elevate your career in platform engineering? We invite applications for a Lead Python Platform Engineer position that requires hands-on experience with Apache Beam, Databricks, and other cutting-edge technologies.

As our ideal candidate, you will be responsible for designing and building a shared component library/SDK for pipelines, defining patterns/templates for Apache Beam pipelines and Databricks jobs, creating pluggable interfaces for multiple teams to swap extractors, OCR providers, and EMR publishers without code rewrites, and implementing robust LLM API rate limiting/governance.

• Architecture and Reuse:
  • Design a shared component library/SDK for ingestion, parsing/OCR, extraction, validation, enrichment, and publishing.
  • Define standardization configuration, packaging, versioning, CI/CD, and documentation for Apache Beam pipelines and Databricks jobs.
  • Create pluggable interfaces for multiple teams to swap extractors, OCR providers, and EMR publishers.
  • Establish a repo strategy for shared/child repos for each use case.

• Performance and Reliability:
  • Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
  • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
  • Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
  • Implement robust LLM API rate limiting/governance: enforce provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
  • Establish SLOs/alerts for throughput, latency, error rates; set up DLQs and recovery patterns.

• Team Enablement:
  • Mentor devs, lead design reviews, codify best practices, write clear docs and examples.
  • Partner with ML engineers on the future LLM/SLM path (evaluation harness, safety/PII, cost/perf).

Our ideal candidate will possess 7+ years of Python experience with strong depth in performance and concurrency, observability expertise, LLM-based solution implementation, API engineering for high-throughput integrations, strong architecture/design skills, and experience building large-scale data pipelines with Apache Beam and/or Spark.

• Expertise You'll Bring:
  • Strong Python foundation with 7+ years of experience.
  • Hands-on experience with Apache Beam, Databricks, and other cutting-edge technologies.
  • Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
  • LLM/SLM integration experience: prompt/eval frameworks, PII redaction/guardrails.
  • Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
  • Security/compliance mindset: HIPAA, secrets management, least-privilege access.

We offer a competitive salary and benefits package, culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications, opportunity to work with cutting-edge technologies, employee engagement initiatives such as project parties, flexible work hours, and Long Service awards, annual health check-ups, insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents.

Persistent Ltd. is dedicated to fostering diversity and inclusion in the workplace. We welcome diverse candidates from all backgrounds and support hybrid work and flexible hours to fit diverse lifestyles.