Senior Data Engineer Architect

1 day ago

Belgaum, Karnataka, India beBeeSoftware Full time ₹ 1,50,00,000 - ₹ 2,10,00,000

Job Title: Senior Software Architect - Data Engineering

Job Summary

We are seeking an experienced Senior Software Architect to lead our data engineering efforts. This individual will be responsible for designing and building a scalable data pipeline using Apache Beam and Databricks.

About the Role

This is a critical position that requires a deep understanding of data processing, architecture, and software engineering principles. The successful candidate will have experience with high-throughput integrations, API engineering, and secure handling of sensitive data.

  • Design and build a shared component library/SDK for pipelines: ingestion, parsing/OCR, extraction (RegEx now, LLM/SLM later), validation, enrichment, publishing.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Create pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.
  • Define repo strategy - shared/child repos for each use case
Responsibilities
  • Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
  • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
  • Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
  • Implement robust LLM API rate limiting/governance: enforce provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
  • Establish SLOs/alerts for throughput, latency, error rates; set up DLQs and recovery patterns.
Requirements

The ideal candidate will possess:

  • 7+ years Python with strong depth in performance and concurrency (asyncio, concurrent.futures, multiprocessing), profiling and memory tuning.
  • Observability expertise: Elastic APM instrumentation and dashboarding; Splunk for logs and correlation; OpenTelemetry familiarity.
  • Must have implemented LLM based solutions and supported them in production
  • API engineering for high-throughput integrations (REST, OAuth2), resilience patterns, and secure handling of sensitive data.
  • Strong architecture/design skills: clean interfaces, packaging shared libs, versioning, CI/CD (GitHub Actions/Azure DevOps), testing.
  • 3+ years building large-scale data pipelines with Apache Beam and/or Spark, including hands-on Databricks experience (Jobs, Delta Lake, cluster tuning).
  • Document processing: OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, text normalization.
  • LLM/SLM integration experience (e.g., OpenAI/Azure AI, local SLMs), prompt/eval frameworks, PII redaction/guardrails.
  • Cloud and tooling: AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker; cost/performance tuning on Databricks.
  • Security/compliance mindset (HIPAA), secrets management, least-privilege access.

  • Data Engineering Specialist

    2 days ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time US$ 15,00,000 - US$ 20,00,000

    Job DescriptionWe are seeking a seasoned Data Engineering Specialist to join our team. As a key member of our data engineering group, you will play a pivotal role in designing, building, and maintaining our data pipeline architecture.Our ideal candidate is an experienced professional with a passion for data engineering, a strong track record of delivering...

  • Senior Data Engineering Leader

    2 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Data Architect Job DescriptionWe are seeking an experienced Data Architect to lead our data engineering initiatives and drive business growth through data-driven decision making.

  • Senior Cloud Data Architect

    4 days ago

    Belgaum, Karnataka, India beBeeEngineering Full time ₹ 1,80,00,000 - ₹ 2,52,00,000

    Cloud Data Engineer LeaderWe are seeking an experienced Cloud Data Engineer Leader to design and deliver scalable data pipelines and analytics platforms.The ideal candidate will lead a team of talented data engineers, overseeing the development of end-to-end data solutions using Azure Data Services and Databricks. They will collaborate with cross-functional...

  • Data Architect

    2 days ago

    Belgaum, Karnataka, India beBeeBusiness Full time ₹ 1,20,00,000 - ₹ 1,50,00,000

    Senior Business Intelligence DeveloperThis role is responsible for designing and developing advanced data warehousing solutions using columnar databases. The successful candidate will have a strong background in ETL processes, proficiency in SQL, and experience with SSIS packages.The ideal candidate will be able to work closely with our team of solution...

  • Senior Backend Software Architect

    7 days ago

    Belgaum, Karnataka, India beBeeBackend Full time US$ 1,50,000 - US$ 2,50,000

    Job Title: Senior Backend Software ArchitectCompany Overview:Coinbase is committed to diversity in its workforce and is proud to be an Equal Opportunity Employer. We are seeking a highly skilled software engineer to join our team.Job Description:We are looking for a talented individual who can architect and develop a platform for financial data. The ideal...

  • Expert Data Architect

    2 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 13,20,000 - ₹ 2,01,60,000

    Data Architect SpecialistWe are seeking an experienced Data Engineer to join our organization. The ideal candidate should have strong expertise in building and automating data pipelines, managing multi-environment deployments, and working with modern DevOps/configuration management tools.

  • Senior Data Engineer

    2 days ago

    Belgaum, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    We are seeking a senior Databricks data engineer to design and develop high-performance data pipelines.Job Overview:The ideal candidate will have expertise in PySpark, CI/CD pipelines, and Terraform for infrastructure as code.Key Responsibilities:Develop scalable data pipelines using Databricks and PySparkImplement and manage data solutions on AWS, including...

  • Chief Data Engineer

    1 day ago

    Belgaum, Karnataka, India beBeeDataCloudArchitect Full time US$ 1,20,000 - US$ 1,60,000

    Data Cloud Architect Job DescriptionAs a key member of our organization, we are seeking an experienced Data Cloud Architect to spearhead data-first initiatives at enterprise scale.This role requires deep knowledge of Salesforce Data Cloud, along with proven leadership skills to guide technical teams and design scalable architectures.Lead end-to-end design,...

  • Data Engineer

    4 weeks ago

    Belgaum, Karnataka, India Petrofac Full time

    Petrofac is a leading international service provider to the energy industry, with a diverse client portfolio including many of the world's leading energy companies.We design, build, manage, and maintain infrastructure for our clients. We recruit, reward, and develop our people based on merit, regardless of race, nationality, religion, gender, age, sexual...

  • Chief Data Architect

    4 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 15,62,500 - ₹ 26,13,500

    We are looking for a skilled Chief Data Architect to join our team. As a key member, you will be responsible for designing and developing data warehouses that meet the needs of our business.This is an exciting opportunity for someone who is passionate about leveraging data structures and algorithms to drive business value. If you have 8+ years of experience...