High-Level Software Architectures for Scalable Data Pipelines

About the Position:

• We are seeking a skilled Python engineer to lead our platform engineering efforts with hands-on experience in Apache Beam, Databricks, and related technologies.

Key Responsibilities:

• Design and build a shared component library/SDK for pipelines: ingestion, parsing/OCR, extraction, validation, enrichment, publishing.
• Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
• Create pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.
• Develop a robust repository strategy - shared/child repos for each use case.

• Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
• Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
• Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
• Implement robust LLM API rate limiting/governance: enforce provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
• Establish SLOs/alerts for throughput, latency, error rates; set up DLQs and recovery patterns.

• Mentor developers, lead design reviews, codify best practices, write clear documentation and examples.
• Partner with ML engineers on the future LLM/SLM path (evaluation harness, safety/PII, cost/perf).

• 7+ years of Python experience with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing), profiling and memory tuning.
• Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
• Must have implemented LLM-based solutions and supported them in production.
• API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
• Strong architecture/design skills: clean interfaces, packaging shared libraries, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
• 3+ years of building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
• Document processing: OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, text normalization.
• LLM/SLM integration experience (e.g., OpenAI/Azure AI, local SLMs), prompt/eval frameworks, PII redaction/guardrails.
• Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
• Security/compliance mindset (HIPAA), secrets management, least-privilege access.

• Competitive salary and benefits package
• Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

• Persistent Ltd. is dedicated to fostering diversity and inclusion in the workplace.
• We welcome diverse candidates from all backgrounds.
• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.

Please visit our careers page for more information.