Python Data Pipeline Specialist

4 days ago

Vizag, Andhra Pradesh, India beBeeEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000
Expert Python Engineer Opportunity

We are seeking a highly skilled engineer to design and build scalable data pipelines using Apache Beam and Databricks.

Key Responsibilities:
  • Develop a shared component library/SDK for ingestion, parsing, extraction, validation, enrichment, and publishing.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Implement pluggable interfaces for multiple teams to swap extractors, OCR providers, and EMR publishers without code rewrites.
  • Design concurrent architectures: asyncio for I/O-bound, ThreadPool/ProcessPool for CPU-bound, batching, rate limiting, retries, etc.
  • Establish Service Level Objectives (SLOs) for throughput, latency, error rates; set up Dead Letter Queues (DLQs) and recovery patterns.
Requirements:
  • 7+ years of experience in Python with strong expertise in performance and concurrency.
  • Observability skills: Elastic APM instrumentation and dashboarding.
  • Experience with Large Language Model (LLM)-based solutions and supporting them in production.
  • Strong architecture/design skills: clean interfaces, packaging shared libraries, versioning, Continuous Integration/Continuous Deployment (CI/CD).
  • Hands-on experience with large-scale data pipelines using Apache Beam and/or Spark.
  • Document processing: Optical Character Recognition (OCR), PDF parsing, text normalization.
About the Role:

This is an excellent opportunity to work on complex projects and collaborate with a team of experts.

Our organization values innovation, collaboration, and professional growth.

About Us:

We are a company that prioritizes diversity, equity, and inclusion. We welcome applications from qualified individuals from all backgrounds.

  • Data Engineering Expert

    4 days ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 22,00,000

    We are seeking an experienced data engineer with strong expertise in Python and AWS services to design, develop, and optimize data pipelines.The ideal candidate will have 5–10 years of experience in designing, developing, and optimizing data pipelines using AWS services such as S3, Glue, Step Functions, and PySpark.Key responsibilities include:Designing...

  • ETL Pipeline Automation Specialist

    5 days ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewThe role of Senior Test Engineer is a critical position in our organization, responsible for ensuring the accuracy, integrity, and performance of ETL Pipelines deployed into Azure cloud environments.This individual will design and develop comprehensive test cases to validate data flows, transformations, and system behaviors using Python scripting...

  • Data Pipeline Specialist

    5 hours ago

    Vizag, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    About Data EngineeringWe are looking for a skilled data engineer to join our team.Job Responsibilities:Design and implement ETL/ELT pipelines using Snowflake Tasks, Streams, and Stored Procedures.Develop JavaScript/SQL stored procedures and UDFs for complex transformations.Write complex SQL queries with window functions, CTEs, and pivots.Manage multi-cluster...

  • Data Engineering Specialist

    5 days ago

    Vizag, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 9,00,000 - ₹ 20,10,000

    Data Engineering SpecialistJob Title: Data Engineering SpecialistWe are seeking a skilled and motivated Data Engineering Specialist to design, develop, and maintain scalable data pipelines that process and transform large volumes of structured and unstructured data.Key Responsibilities:Design and Implement Scalable Data PipelinesBuild and maintain ETL/ELT...

  • Python Programmer

    15 hours ago

    Vizag, Andhra Pradesh, India beBeeData Full time ₹ 8,00,000 - ₹ 12,50,000

    Senior Clinical Data Specialist - Python ProgrammingThe role of the Senior Clinical Data Specialist is to provide comprehensive clinical data expertise as part of a team to develop and maintain Laboratory Data Management tasks for studies awarded to a leading contract research organization.Key Responsibilities:Demonstrate in-depth knowledge and expertise in...

  • Data Ingestion Specialist

    7 days ago

    Vizag, Andhra Pradesh, India beBeeDataIngestion Full time US$ 90,000 - US$ 1,40,000

    Job DescriptionWe are seeking an experienced Data Ingestion Specialist to join our team. The ideal candidate will have a strong understanding of data ingestion processes and be proficient in designing, developing, and optimizing data pipelines for integrating multiple sources into Databricks.The successful candidate will have hands-on experience with data...

  • Data Architecture Specialist

    6 hours ago

    Vizag, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Architecture SpecialistSynergy exists between creativity and innovative technology to deliver leading-edge digital solutions. We seek a seasoned professional with expertise in digital channel activities, data engineering, and cloud technologies, particularly AWS.The ideal candidate will possess strong knowledge of AWS, Python, and work...

  • Data Infrastructure Specialist

    6 hours ago

    Vizag, Andhra Pradesh, India beBeeDataInfrastructure Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Infrastructure SpecialistWe are seeking a skilled professional to design, develop, and maintain scalable data infrastructure that powers insights and analytics for our slot games.Key Responsibilities:Develop, maintain, and optimize large-scale data pipelines for slot game and player data.Ensure high-quality, reliable, and accurate data across...

  • Big Data Specialist

    2 days ago

    Vizag, Andhra Pradesh, India beBeeBigData Full time ₹ 15,00,000 - ₹ 25,00,000

    Senior Big Data SpecialistAs a key contributor to our product engineering team, you will design, develop and maintain large-scale data pipelines for ingestion, transformation and storage.This role requires strong expertise in SQL query optimization, stored procedures and complex joins. Proficiency in programming languages like Python, Java or Scala is also...

  • Python Architect

    5 hours ago

    Vizag, Andhra Pradesh, India Sails Software Solutions Full time US$ 1,50,000 - US$ 2,00,000 per year

    Python ArchitectPrimary Requirements:• Experience: >12+yrs of experience working on python & automation project• Expert Proficiency in Python Programming Language: Deep understanding of Python internals, design patterns, and best practices.• Architectural Design and System Planning: Proven ability to design scalable, maintainable, and efficient...