Senior Data Pipeline Engineer

5 days ago

Ajmer, Rajasthan, India beBeeSoftware Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

About the Role:

We are seeking a highly experienced software engineer with expertise in designing and developing scalable data pipelines.

Key Responsibilities:
  • Architecture and Reuse:
    • Design and develop reusable components for ingestion, parsing, extraction, validation, enrichment, and publishing of data.
    • Define patterns/templates for data processing and standardize configuration, packaging, versioning, CI/CD, and documentation.
    • Create pluggable interfaces so multiple teams can swap extractors, OCR providers, and publishers without code rewrites.
    • Develop a repository strategy for shared and child repositories.
  • Performance and Reliability:
    • Own end-to-end profiling and tuning of services using cProfile, py-spy, line_profiler, memory analysis, CPU vs I/O analysis.
    • Instrument services with Elastic APM and correlate traces/metrics with Splunk logs; build dashboards and runbooks.
    • Implement concurrency best practices: asyncio for I/O-bound tasks, ThreadPool/ProcessPool for CPU-bound tasks, batching, rate limiting, retries, etc.
    • Implement robust API rate limiting/governance: enforce provider TPM and concurrency caps, request queueing/token budgeting, and emit APM/Splunk metrics (throttle rate, queue depth, cost per job) with alerts.
    • Establish service-level objectives/alerts for throughput, latency, error rates; set up dead-letter queues and recovery patterns.
  • Team Enablement:
    • Mentor developers, lead design reviews, codify best practices, write clear documentation and examples.
    • Partner with machine learning engineers on the future path (evaluation harness, safety/PII, cost/perf).

  • Data Pipeline Specialist

    6 days ago

    Ajmer, Rajasthan, India beBeeData Full time ₹ 18,95,000 - ₹ 25,10,000

    Our organization seeks a Senior Data Engineer to contribute to the design and development of scalable, high-performance data pipelines using Databricks and PySpark.This role is part of a dynamic team that collaborates with data scientists, analysts, and other engineering teams to support end-to-end data solutions. As a Senior Data Engineer, you will be...

  • Data Pipeline Specialist

    7 days ago

    Ajmer, Rajasthan, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job SummaryWe are seeking a highly skilled Data Engineer to design, build and maintain large-scale data pipelines. This individual will work with streaming data, collaborate with data scientists and stakeholders to deliver high-quality solutions.Key Responsibilities:Design and optimize complex data pipelines using Spark and ETL workflows.Manage and process...

  • Data Pipeline Engineer

    1 day ago

    Ajmer, Rajasthan, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Description">As a skilled ETL Developer, you will play a crucial role in designing and implementing efficient data pipelines using IBM DataStage and AWS Glue/Lambda.Key responsibilities include:Designing and developing ETL pipelines for ingesting data from various sources like flat files, APIs, Oracle, and DB2.Building and optimizing data flows for...

  • Senior Data Engineering Leader

    5 days ago

    Ajmer, Rajasthan, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Summary:We are seeking a Senior Data Engineer to lead the development and maintenance of our data pipeline architecture. This is a critical role that will play a pivotal part in shaping our data strategy.Design, build, and maintain highly scalable data pipelines to support advanced analytics and machine learning.Collaborate with cross-functional teams to...

  • Software Engineer

    2 days ago

    Ajmer, Rajasthan, India beBeeDataScientist Full time ₹ 12,00,000 - ₹ 25,00,000

    Python DeveloperWe are seeking a highly skilled Python developer to spearhead the design and implementation of automated data pipelines using cloud-based services.Key Responsibilities:Design and implement data pipelines leveraging cloud-based technologiesDevelop analytical algorithms to drive business insightsBuild REST APIs for seamless data...

  • Cloud Data Pipeline Specialist

    23 hours ago

    Ajmer, Rajasthan, India beBeeDataEngineering Full time ₹ 90,00,000 - ₹ 1,20,00,000

    Seeking a Data Engineering expert to build and maintain robust data pipelines using cloud-based technologies.Key Responsibilities:Design, develop and maintain scalable data pipelines using cloud services.Implement data ingestion, transformation and storage solutions.Ensure data quality, integrity and security across all data systems.Monitor and troubleshoot...

  • Chief Data Pipeline Architect

    3 days ago

    Ajmer, Rajasthan, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Summary:We are seeking a highly motivated and experienced Data Engineer to design, build, and maintain robust data pipelines. The ideal candidate will have deep expertise in cloud-based data processing technologies using GCP, AWS, or Azure.Responsibilities:Design, develop, and maintain scalable data pipelines using cloud services (GCP, AWS, or...

  • Principal Data Pipeline Specialist

    23 hours ago

    Ajmer, Rajasthan, India beBeeDataEngineering Full time ₹ 30,00,000 - ₹ 50,00,000

    Job SummaryDesign and build production-level data pipelines to handle large volumes of structured and unstructured data.Ensure the reliability, scalability, and high availability of end-to-end ETL/ELT processes.Work with AWS services (S3, EC2, RDS, DynamoDB, Athena, Redshift) to power large-scale data workflows.Collaborate closely with AI/ML teams to support...

  • Data Engineer – Pipeline Specialist

    4 days ago

    Ajmer, Rajasthan, India beBeedataengineer Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    ">Design and Develop Data Pipelines with Python">

  • Data Pipeline Specialist

    1 week ago

    Ajmer, Rajasthan, India beBeeEtl Full time ₹ 15,00,000 - ₹ 20,00,000

    ETL DeveloperWe are seeking a skilled ETL professional to join our data engineering team. The ideal candidate will have expertise in designing, developing and maintaining scalable data pipelines using IBM DataStage and AWS Glue.The successful applicant will collaborate with architects, business analysts and data modelers to ensure timely delivery of critical...