Platform Architecture and Engineering Lead

About the Role:

We are seeking a highly skilled Python Engineer to lead our platform architecture and performance efforts. This individual will be responsible for designing and building a shared component library/SDK for pipelines, defining patterns/templates for Apache Beam pipelines and Databricks jobs, and creating pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.

Key Responsibilities:

• Architecture and Reuse:
  • Design and build a shared component library/SDK for pipelines: ingestion, parsing/OCR, extraction (RegEx now, LLM/SLM later), validation, enrichment, publishing.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Create pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.
  • Define repository strategy - shared/child repositories for each use case.
• Performance and Reliability:
  • Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
  • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
  • Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
  • Implement robust LLM API rate limiting/governance: enforce provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
  • Establish SLOs/alerts for throughput, latency, error rates; set up Dead Letter Queues (DLQs) and recovery patterns.
• Team Enablement:
  • Mentor developers, lead design reviews, codify best practices, write clear documentation and examples.
  • Partner with machine learning engineers on the future LLM/SLM path (evaluation harness, safety/PII, cost/perf).

Requirements:

• 7+ years of experience in Python development with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing), profiling and memory tuning.
• Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
• Must have implemented LLM-based solutions and supported them in production.
• API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
• Strong architecture/design skills: clean interfaces, packaging shared libraries, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
• 3+ years of experience building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
• Document processing: OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, text normalization.
• LLM/SLM integration experience (e.g., OpenAI/Azure AI, local SLMs), prompt/eval frameworks, PII redaction/guardrails.
• Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
• Security/compliance mindset (HIPAA), secrets management, least-privilege access.

Benefits:

• Competitive salary and benefits package.
• Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications.
• Opportunity to work with cutting-edge technologies.
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards.
• Annual health check-ups.
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents.

Work Environment:

Persistent Ltd. is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.