Senior Python Pipeline Architect

5 days ago

Mangalore, Karnataka, India beBeeEngineer Full time ₹ 1,80,00,000 - ₹ 2,40,00,000
Senior Python Engineer Position

About This Role:

We are seeking a highly skilled Senior Python Engineer to lead our team in developing scalable data pipelines using Apache Beam and Databricks.

Key Responsibilities
  • Pipeline Architecture
    • Create a shared component library/SDK for ingestion, parsing/extraction, validation, enrichment, and publishing.
    • Define standard patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
    • Design pluggable interfaces for multiple teams to swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.
    • Determine the repository strategy for each use case.
  • Performance Optimization
    • Own end-to-end profiling and tuning: cProfile/py-spy/line_profiler, memory (tracemalloc), CPU vs I/O analysis.
    • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
    • Implement concurrency best practices: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
    • Enforce robust LLM API rate limiting/governance: provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
    • Establish Service Level Objectives/alerts for throughput, latency, error rates; set up Dead Letter Queues and recovery patterns.
  • Team Enablement
    • Mentor developers, lead design reviews, codify best practices, write clear documentation and examples.
    • Partner with Machine Learning engineers on the future LLM/SLM path (evaluation harness, safety/PII, cost/perf).

  • Data Pipeline Architect

    4 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data EngineerAbout the Role:We are seeking a highly skilled Senior Data Engineer to join our team. As a key member of our engineering group, you will be responsible for designing, developing, and implementing large-scale data processing systems.Key Responsibilities:Design and develop scalable ETL workflows using Python and Pandas.Implement and manage...

  • AI Pipeline Architect

    16 hours ago

    Mangalore, Karnataka, India beBeeLeader Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job DescriptionWe are seeking a Senior AI Engineer to oversee the orchestration of our end-to-end pipeline, which combines cutting-edge models, agile workflows, and automation.This role requires in-depth expertise in speech and audio AI, agile pipeline orchestration, and scalable system design knowledge.The ideal candidate has a strong track record of...

  • Senior Data Architect

    4 days ago

    Mangalore, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Posting:We seek a highly skilled Senior Data Architect to join our organization. This individual will be responsible for designing and implementing scalable data systems following Medallion Architecture and best practices.Key Responsibilities:Data Ingestion and Transformation: Develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF),...

  • Data Pipeline Architect

    5 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 30,00,000

    Senior Data Engineer">">Our company is seeking an experienced Senior Data Engineer to join our team. This role will play a critical part in building scalable data pipelines and integrating healthcare data.">

  • Chief Data Pipeline Architect

    10 hours ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Engineer PositionWe are seeking a highly skilled Senior Data Engineer to join our team at a cutting-edge organization.This role is critical to the development of new data products and the modernization of existing platforms in cloud environments, especially AWS.Key ResponsibilitiesDesign, build, and maintain scalable data pipelines and data...

  • Data Architect

    3 days ago

    Mangalore, Karnataka, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job DescriptionAs a senior Data Architect, you will be responsible for designing and implementing a scalable data warehouse architecture on PostgreSQL, leveraging open-source technologies. You will work closely with cross-functional teams to deliver high-quality, reliable data solutions that meet business and analytics requirements.Key...

  • Senior Data Pipeline Architect

    2 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Key ResponsibilitiesWe're looking for a skilled professional to lead the development of scalable data pipelines.Design and Implement Efficient ETL ProcessesCollaborate with Data TeamsImplement Data Governance and Security StandardsOptimize Data Workflows for Scalability and PerformanceRequired Skills & QualificationsTo succeed in this role, you'll...

  • Senior Data Architect

    10 hours ago

    Mangalore, Karnataka, India beBeeDataArchitect Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    We are seeking a skilled Senior Data Architect to join our team.The successful candidate will be responsible for designing, building and maintaining efficient data pipelines. This includes collaborating closely with cross-functional teams to ensure data is clean, reliable and accessible for analytics, reporting and product innovation.This is a senior role...

  • Python Software Architect

    10 hours ago

    Mangalore, Karnataka, India beBeeSoftware Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Summary:We are seeking a highly skilled Python Software Architect to lead our software development efforts.The ideal candidate will have in-depth knowledge and experience with Python, Django, Angular, Ansible, Middleware, and Domain knowledge of Frontend and Backend development and integration.Key Responsibilities:Design and implement scalable software...

  • Senior Data Pipeline Engineer

    16 hours ago

    Mangalore, Karnataka, India beBeeDataSolutions Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Job Title: Data Solutions ArchitectAs a highly skilled data solutions architect, you will play a pivotal role in designing and developing scalable data pipelines and ETL processes on Databricks. Your expertise will be instrumental in leading the architecture and implementation of secure, high-performance data solutions.Key Responsibilities:Develop and...