Python Architectural Leader

About This Position:

We are seeking an experienced Platform Engineer to lead our Python team in architecture and performance. The ideal candidate will have hands-on experience with Apache Beam, Databricks, and other related technologies.

• Role: Lead Python Platform Engineer – Architecture & Performance
• Location: All Locations
• Experience: 3 to 7 Years
• Job Type: Full Time Employment

Our platform engineer will be responsible for designing and building a shared component library/SDK for pipelines, defining patterns/templates for Apache Beam pipelines and Databricks jobs, creating pluggable interfaces for multiple teams, and establishing SLOs/alerts for throughput, latency, error rates.

• Design and build a shared component library/SDK for pipelines: ingestion, parsing/OCR, extraction (RegEx now, LLM/SLM later), validation, enrichment, publishing.
• Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
• Create pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.
• Define repo strategy - shared/child repos for each use case.
• Establish SLOs/alerts for throughput, latency, error rates; set up DLQs and recovery patterns.

The successful candidate will possess strong expertise in Python with performance and concurrency, observability, API engineering, and large-scale data pipeline development using Apache Beam and/or Spark. Experience with LLM/SLM integration, document processing, cloud tooling, and security compliance is also required.

• 7+ years Python with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing), profiling and memory tuning.
• Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
• Must have implemented LLM based solutions and supported them in production
• API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
• Strong architecture/design skills: clean interfaces, packaging shared libs, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
• 3+ years building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
• Document processing: OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, text normalization.
• LLM/SLM integration experience (e.g., OpenAI/Azure AI, local SLMs), prompt/eval frameworks, PII redaction/guardrails.
• Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
• Security/compliance mindset (HIPAA), secrets management, least-privilege access.

We offer a competitive salary and benefits package, culture focused on talent development, opportunity to work with cutting-edge technologies, employee engagement initiatives, annual health check-ups, and insurance coverage.

• Competitive salary and benefits package
• Culture focused on talent development with quarterly promotion cycles and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

We value diversity and inclusion and welcome applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We support hybrid work and flexible hours to fit diverse lifestyles, and our office is accessibility-friendly.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.