Data Pipeline Engineer

1 week ago

Kota, Rajasthan, India beBeebackend Full time ₹ 15,00,000 - ₹ 20,00,000
Senior Backend Data & Integration Engineer

A seasoned software engineer is required to design and implement data pipelines, integrate external systems and interfaces, and develop crawling/fetch pipelines.

Key Responsibilities:

  1. Development of efficient crawling/fetch pipelines using API-first approach with playwright/requests where permitted.
  2. Parsing and normalization of job postings & CVs, deduplication/delta logic including hash seen repost heuristics.
  3. Embeddings and similarity search utilizing Azure OpenAI and vector persistence in pgvector.
  4. Integrations: HR4YOU via API/webhooks/CSV import, SerpAPI, BA job board, email/SMTP.
  5. Batch/stream processing leveraging Azure Functions/container jobs, retry/backoff, dead-letter queues.
  6. Telemetry for data quality including freshness, duplicate rate, coverage, cost per 1,000 items.
  7. Collaboration with frontend team for exports (CSV/Excel, presigned URLs) and admin configuration.

Requirements:

  1. 4+ years of backend/data engineering experience.
  2. Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), solid TypeScript for smaller services/SDKs.
  3. Azure expertise: Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics.
  4. Messaging knowledge: Service Bus/Queues, idempotence & exactly-once semantics, pragmatic approach.
  5. Databases: PostgreSQL, pgvector, query design & performance tuning.
  6. Clean ETL/ELT patterns, testability (pytest), observability (OpenTelemetry).

Nice-to-Have:

  1. NLP/IE experience (spaCy/regex/rapidfuzz), document parsing (pdfminer/textract).
  2. Licence compliant data retrieval, captcha/anti-bot strategies.
  3. API-first working method, clean code, trunk-based development, mandatory code reviews.
  4. Tools/stack: GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo, Jira/Linear, Notion/Confluence.
Essential Skills & Qualifications

We are looking for a skilled software engineer with a strong background in backend/data engineering.

Desired Qualifications:

  • Backend programming languages: Python, TypeScript.
  • Database management: PostgreSQL, pgvector.
  • Messaging systems: Service Bus/Queues.
  • Azure platform: Functions/Container Apps, Storage/Blob, Key Vault.

Bonus Points:

  • NLP/IE skills.
  • Data retrieval & anti-bot strategies.

  • Data Pipeline Architect

    2 weeks ago

    Kota, Rajasthan, India beBeeAirflow Full time ₹ 15,00,000 - ₹ 19,00,000

    Expert Data Pipeline DeveloperAs a seasoned data engineer, you will be responsible for designing and implementing scalable and reliable data pipelines using Airflow.Develop complex workflows using Airflow, ensuring high-quality data flow and meeting business requirements.Collaborate with cross-functional teams to integrate data pipelines with business...

  • Data Pipeline Architect

    2 weeks ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 18,00,000

    Cloud Data Engineer OpportunityWe are seeking a highly skilled AWS Cloud Data Engineer to design, build, and maintain scalable data pipelines on AWS. The ideal candidate will have a solid background in cloud-based data solutions, object-oriented programming, reusable code design, and Python development.This role involves building, optimizing, and maintaining...

  • GCP Data Engineer

    3 weeks ago

    Kota, Rajasthan, India CustomerLabs 1P Data OPs Full time

    Position Overview:"Yesterday is history, tomorrow is a mystery, but today is a gift. That's why we call itthe present." - Master OogwayJoin CustomerLabs' dynamic data team as a Data Engineer and play a pivotal role intransforming raw marketing data into actionable insights that power our digitalmarketing platform. As a key member of our data infrastructure...

  • Data Pipeline Architect

    1 week ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Scalable Data EngineerWe are seeking a highly motivated and talented data engineer to join our team. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.Key Responsibilities:Design, develop, and maintain ETL pipelines using IBM...

  • Lead Data Pipeline Developer

    2 weeks ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 1,80,00,000

    **Job Role - Data Engineer With Fabric**We are seeking a highly skilled data engineer to design, develop and maintain our data pipelines and infrastructure.As a key member of our team, you will work closely with data scientists, analysts, and other stakeholders to ensure data is accurate, timely and accessible.Design, develop and implement data pipelines...

  • Lead Data Pipeline Architect

    2 weeks ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 19,00,000

    Data Engineer Role">This position involves designing, building, and optimizing data pipelines to ensure that data is clean, secure, and accessible for analytics and operations.">As a member of our team, you will collaborate with talented individuals to solve real-world data challenges and make a direct impact on business decisions.">About Us">We are a...

  • Senior Financial Data Pipeline Specialist

    1 week ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Large Scale Data Engineering Opportunity\We are seeking a skilled data engineer to work on a large-scale financial institution project.\The ideal candidate will have experience designing and deploying data pipelines, orchestrating workflows, processing data, managing datasets, and integrating with APIs and CI/CD pipelines.\\Must have 6+ years of experience...

  • Data Architect for Cloud-Based Data Pipelines

    1 week ago

    Kota, Rajasthan, India beBeeETL Full time ₹ 15,00,000 - ₹ 20,00,000

    We are seeking a highly experienced Senior Informatica Engineer to design and develop cloud-based ETL pipelines using Azure services and integrate data into AWS Datalake or other modern data platforms.

  • Transformative Data Architect

    1 week ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 1,60,00,000

    As a key member of our data engineering team, you will play a pivotal role in designing and developing scalable data pipelines and transformation frameworks that enable analytics, reporting, and business intelligence capabilities across the enterprise.About the RoleWe are seeking an experienced Data Engineer with hands-on expertise in the Microsoft Azure...

  • AWS Cloud Data Pipeline Architect

    2 weeks ago

    Kota, Rajasthan, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    As a Data Engineering Specialist, you will play a key role in designing and implementing data pipelines using AWS services.Key Responsibilities:Design and develop scalable data pipelines using AWS services including Glue, PySpark, SQL, Databricks, and Python.Develop and maintain applications using Python and PySpark for data manipulation, transformation, and...