Data Pipeline Architect

2 days ago

Vijayawada, Andhra Pradesh, India beBeeEngineer Full time ₹ 20,00,000 - ₹ 25,00,000
Job Description

We are seeking a seasoned Python Platform Engineer to lead the design and development of high-performance data pipelines. The ideal candidate will have extensive experience with Apache Beam, Databricks, and other related technologies.

The successful candidate will be responsible for designing and building shared component libraries, standardizing configuration, packaging, versioning, CI/CD, and documentation. They will also create pluggable interfaces to enable multiple teams to swap extractors, OCR providers, and EMR publishers without code rewrites.

  • Design and build shared component libraries/SDKs for ingestion, parsing/OCR, extraction, validation, enrichment, and publishing.
  • Define patterns/templates for Apache Beam pipelines and Databricks jobs; standardize configuration, packaging, versioning, CI/CD, and documentation.
  • Create pluggable interfaces so multiple teams can swap extractors (Regex/LLM), OCR providers, and EMR publishers without code rewrites.

Requirements

To be successful in this role, you will need:

  • 7+ years of experience with Python, with a strong focus on performance and concurrency.
  • Strong expertise in observability, including Elastic APM instrumentation and dashboarding, Splunk for logs and correlation, and OpenTelemetry familiarity.
  • Experience implementing LLM-based solutions and supporting them in production.
  • API engineering skills for high-throughput integrations, resilience patterns, and secure handling of sensitive data.
  • Strong architecture/design skills, including clean interfaces, packaging shared libs, versioning, CI/CD, and testing.
  • Hands-on experience with Apache Beam, Databricks, and/or Spark, including large-scale data pipeline development.
  • Document processing expertise, including OCR (Tesseract, AWS Textract, Azure Form Recognizer), PDF parsing, and text normalization.
  • Experience with cloud and tooling platforms, including AWS/Azure/GCP, Dataflow/Flink, Terraform, Docker, and cost/performance tuning on Databricks.
  • A security/compliance mindset, with expertise in secrets management and least-privilege access.

What We Offer

In return for your expertise, we offer:

  • Competitive salary and benefits package.
  • A culture focused on talent development, with quarterly promotion cycles and company-sponsored higher education and certifications.
  • The opportunity to work with cutting-edge technologies.
  • Employee engagement initiatives, such as project parties, flexible work hours, and Long Service awards.
  • Annual health check-ups.
  • Insurance coverage, including group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents.

Diversity, Equity, and Inclusion

We are committed to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, regardless of gender or gender preference, and welcome diverse candidates from all backgrounds.

  • Senior Data Architect Position

    2 days ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 20,20,000

    Data ArchitectWe are seeking an experienced data architect to lead our data engineering initiatives. As a key member of our team, you will be responsible for designing and implementing large-scale data pipelines using Python and AWS services.

  • Data Pipeline Specialist

    4 days ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Cloud Data EngineerWe are seeking a skilled Cloud Data Engineer with hands-on experience in Snowflake, AWS (Lambda, Glue), DBT, and SQL to design, build, and optimize scalable data pipelines. The ideal candidate will be responsible for enabling seamless data integration, transformation, and analytics to support business intelligence and advanced analytics...

  • Designing Scalable Data Pipelines

    2 days ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 40,00,000

    Job Title:Senior Data EngineerAbout the RoleWe are seeking a seasoned data engineer to design, build, and operate scalable data pipelines that process clinical encounter data, retrieve associated medical documents, and deliver validated information to our machine learning systems for inference.The ideal candidate will have expertise in building event-driven...

  • Data Architect

    1 day ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Architect We're seeking a skilled Data Scientist, Data Visualization Expert (Python), and Data Engineer to join our team on a remote project. This role is focused on creating high-quality datasets and visualizations for Large Language Model training. Key Responsibilities: Design and create advanced data visualizations using Python libraries like pandas,...

  • Master Data Architect

    22 hours ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    About this Position:We are seeking a skilled Data Architect to develop and implement scalable data systems. As a member of our team, you will design, build, and maintain reliable data infrastructure that supports real-time and batch data processing.You will collaborate with cross-functional teams, including data scientists, analysts, and software engineers,...

  • Data Scientist/ML Engineer

    6 days ago

    Vijayawada, Andhra Pradesh, India Quant-data Full time

    We're Hiring: Machine Learning Engineer / Data Engineer (Remote | Full-Time) Build AI-powered credit decisioning systems on Microsoft AzureWe're looking for a Machine Learning Engineer / Data Engineer with 5+ years of experience to join our AI-driven credit lending platform team. In this role, you'll design and deploy scalable ML solutions that power loan...

  • Advanced Data Engineering Professional

    2 days ago

    Vijayawada, Andhra Pradesh, India beBeeData Full time US$ 1,50,000 - US$ 2,50,000

    Job DescriptionAs a Senior Data Engineer, you will be responsible for designing and building scalable data pipelines, collaborating with cross-functional teams to ensure efficient data flow. You will lead the optimization of data delivery pipelines, architect high-throughput data pipelines and infrastructure, partner with data architects and analysts to...

  • ETL Specialist

    4 days ago

    Vijayawada, Andhra Pradesh, India beBeeETL Full time ₹ 15,00,000 - ₹ 20,00,000

    We are seeking a skilled professional to fill the role of ETL Developer, specializing in Talend Open Studio and cloud data pipelines.Position Overview:  The selected candidate will design and optimize complex ETL pipelines, leveraging their expertise in Talend Data Integration, BigQuery, PostgreSQL, Python, and Google Cloud Platform.  Strong...

  • Expert Large Scale Data Pipeline Developer

    23 hours ago

    Vijayawada, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job DescriptionWe are seeking an experienced Data Engineer to design, build, and optimize large-scale data pipelines. The ideal candidate will have expertise in Python, SQL, Linux/Bash, PySpark, AWS, and Databricks.The primary focus of this role is to develop, maintain, and optimize ETL/ELT pipelines using PySpark, AWS, and Databricks. Additionally, the...

  • Chief Data Insights Architect

    7 days ago

    Vijayawada, Andhra Pradesh, India beBeeDataInsights Full time ₹ 17,50,000 - ₹ 25,10,000

    Unlock the full potential of your organization with our Chief Data Insights Architect role.About the RoleLeverage data, analytics, and artificial intelligence (AI) technologies to drive business growth and innovation.Develop and deploy data-driven solutions using machine learning algorithms and statistical modeling techniques.Collaborate with...