Data Pipeline Engineer

14 hours ago

Vellore, Tamil Nadu, India beBeeBackend Full time ₹ 15,00,000 - ₹ 18,00,000

Job Opportunity: Data Engineering Specialist

We are seeking a skilled data engineer to build and maintain our data pipelines, ensuring seamless connections between external systems and interfaces.

Key Responsibilities:

  • Develop efficient crawling/fetch pipelines using API-first approach with playwright/requests where permitted.
  • Parsing and normalization of job postings and CVs, deduplication/delta logic including seen hash and repost heuristics.
  • Embeddings/similarity search controlling Azure OpenAI and vector persistence in pgvector.
  • Integrate HR4YOU, SerpAPI, BA job board, email, and SMTP services.
  • Design batch/stream processing solutions using Azure Functions/container jobs, retry/backoff, and dead-letter queues.
  • Implement telemetry for data quality, including freshness, duplicate rate, coverage, and cost per 1,000 items.

Requirements:

  • 4+ years of backend/data engineering experience.
  • Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), TypeScript, and Azure technologies.
  • Knowledge of databases, PostgreSQL, pgvector, query design, and performance tuning.
  • Clean ETL/ELT patterns, testability (pytest), and observability (OpenTelemetry).

Benefits:

  • Collaborate with frontend team for exports (CSV/Excel) and admin configuration.
  • Work on a dynamic and challenging project.

Additional Information:

  • NLP/IE experience with spaCy/regex/rapidfuzz, document parsing with pdfminer/textract.
  • Experience with license/ToS-compliant data retrieval and captcha/anti-bot strategies.

  • Data Pipeline Architect

    3 days ago

    Vellore, Tamil Nadu, India beBeeDataPipeline Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Pipeline ArchitectDesign and develop scalable, reliable data platforms to support business objectives.We are looking for a seasoned Data Architect to design, build, and maintain complex data pipelines. You will collaborate with stakeholders to gather requirements and translate business needs into robust technical solutions.Develop data pipelines using...

  • Data Pipelining Specialist

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeETLPipeliningEngineer Full time ₹ 18,00,000 - ₹ 22,00,000

    Job Title: Data Pipelining Engineer.We are seeking a skilled Data Pipelining Engineer to join our data engineering team. The successful candidate will be responsible for designing, developing, and maintaining scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.Key Responsibilities: Design and develop...

  • Expert Data Pipeline Specialist

    1 week ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job TitleWe are seeking an experienced Freelance Data Engineer to design, build, and optimize large-scale data pipelines. The consultant will collaborate with internal teams and clients to ensure robust, scalable, and high-performance data solutions.Key Responsibilities:Design, develop, and maintain ETL/ELT pipelines using PySpark, AWS, and...

  • Azure Data Engineer: Expertise in Data Pipeline Design and Implementation

    1 week ago

    Vellore, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 1,50,00,000

    Job TitleAzure Data Engineer: Expertise in Data Pipeline Design and ImplementationJob DescriptionWe are seeking a skilled professional with expertise in designing, implementing, and maintaining data pipelines using ADF, ADB, PySpark, and SQL.The ideal candidate will have strong passion for data engineering and keen eye for detail.Required Skills and...

  • Chief Data Pipeline Architect

    1 week ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Azure Data EngineerWe are seeking a skilled Azure Data Engineer to fill this key role in our organization.Key Responsibilities:Design and implement data pipelines using PySpark, SQL, and Azure Data Factory (ADF)Collaborate with stakeholders to deliver scalable data-driven outcomesWork in Agile teams to handle data validation, transformation, and...

  • Building Scalable Data Pipelines

    1 week ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 16,00,000 - ₹ 22,00,000

    About this JobWe are seeking a Senior Data Engineer to lead our data infrastructure initiatives. This critical role involves designing and operating scalable data pipelines that support real-time clinical workflows.Key Responsibilities:Create event-driven data pipelines integrating healthcare data from various sources.Collaborate with software engineers and...

  • Building Scalable Data Pipelines

    1 week ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineering Opportunities at ScaleAbout the RoleWe are seeking a seasoned Data Engineer to join our team and help drive business growth through data-driven insights. This is an exceptional opportunity to leverage your technical expertise and collaborate with talented professionals to build and optimize scalable data pipelines.Key ResponsibilitiesCreate...

  • Senior Cloud Data Pipeline Specialist

    6 days ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 2,50,00,000

    Cloud-Based Data EngineerWe are seeking an experienced Data Engineer to design, build, and maintain robust data pipelines in a cloud-based environment.Key Responsibilities:Design, develop, and maintain scalable data pipelines using cloud services (GCP, AWS, or Azure).Implement data ingestion, transformation, and storage solutions.Ensure data quality,...

  • Senior Data Pipeline Specialist

    1 week ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,09,00,000

    About Us:We're a leading provider of telenutrition and foodcare solutions. Our platform empowers healthier food choices, drives lasting behavior change, and delivers long-term health outcomes.Our highly personalized digital platform guides users on a tailored journey to eating well while saving time and money.We integrate dietary assessments and nutrition...

  • Data Pipeline Specialist

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeEtl Full time ₹ 15,90,000 - ₹ 19,90,000

    We are seeking a skilled ETL developer to join our team. The ideal candidate will have experience in designing and optimizing complex data pipelines, debugging, and deployment using Talend.**Position Details:**- **Role:** ETL Developer- **Experience:** 8 to 10 years- **Work Timings:** Flexible schedule with evening shifts- **Duration:** 3 Months (Contract)-...