Data Pipeline Engineer

2 weeks ago

Allahabad, Uttar Pradesh, India beBeeBackend Full time ₹ 10,00,000 - ₹ 15,00,000
Senior Backend Data & Integration Engineer Position

We are seeking an experienced and skilled Senior Backend Data & Integration Engineer to join our team. As a key member of our engineering team, you will play a crucial role in designing and implementing robust data pipelines.

The ideal candidate will have 4-8 years of experience in backend/data engineering with expertise in Python, TypeScript, Azure, and PostgreSQL.

  • Develop crawling/fetch pipelines using API-first approach with Playwright/Requests.
  • Parsing/normalization of job postings & CVs, deduplication/delta logic using seen hash, repost heuristics.
  • Embeddings/similarity search using Azure OpenAI and vector persistence in pgvector.
  • Integrations: HR4YOU, SerpAPI, BA job board, email/SMTP.
  • Batch/stream processing using Azure Functions/container jobs, retry/backoff, dead-letter queues.
  • Telemetry for data quality (freshness, duplicate rate, coverage, cost per 1,000 items).
  • Collaborate with FE for exports (CSV/Excel, presigned URLs) and admin configuration.
Requirements

To be successful in this role, you must possess the following qualifications:

  • 4+ years of backend/data engineering experience.
  • Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), solid TypeScript for smaller services/SDKs.
  • Azure: Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics.
  • Messaging: Service Bus/Queues, idempotence & exactly-once semantics, pragmatic approach.
  • Databases: PostgreSQL, pgvector, query design & performance tuning.
  • Clean ETL/ELT patterns, testability (pytest), observability (OpenTelemetry).
Desirable Skills

While not mandatory, the following skills would be highly beneficial:

  • NLP/IE experience (spaCy/regex/rapidfuzz), document parsing (pdfminer/textract).
  • Experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies (legally compliant).
  • Working method: API-first, clean code, trunk-based development, mandatory code reviews.
  • Tools/stack: GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, Notion/Confluence.

  • Senior Data Pipeline Engineer

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Seeking a skilled and driven Senior ETL Developer to join our data engineering team. The successful candidate will design and implement scalable and efficient data pipelines using cutting-edge technologies.">Design and develop large-scale data pipelines using IBM DataStage, AWS Glue, and SnowflakeCollaborate with cross-functional teams to ensure seamless...

  • Strategic Data Pipeline Developer

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data Engineer Job DescriptionOur organization is seeking a skilled Senior Data Engineer to design, develop, and maintain data pipelines that extract data from Oracle Symphony via APIs, process, and store it in the Databricks Lakehouse platform. The successful candidate will integrate the data into Oracle EPM (Enterprise Performance Management) to...

  • Senior Data Pipeline Architect

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineer RoleWe are seeking a highly skilled and experienced Data Engineer to join our team. The ideal candidate will have a strong background in designing and developing scalable data pipelines that process massive datasets efficiently.Key Responsibilities:Data Pipeline Design & Development: Utilize Python and leverage GCP services such as Vertex AI,...

  • Chief Data Pipeline Architect

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    About Our Data Engineering RoleWe are seeking a highly skilled Senior Data Engineer with expertise in building scalable, event-driven data pipelines and integrating healthcare data.Design and implement efficient data pipelines to support real-time clinical workflows.Integrate with Electronic Health Record (EHR) systems using standards like FHIR and...

  • Designing Robust Data Pipelines

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 25,00,000

    We are seeking a skilled professional to take on the role of Data Pipeline Engineer.Key Responsibilities:Create and maintain efficient data processing systems using Python, SQL, PySpark, and bash scripts.Develop and maintain data pipelines for structured and unstructured data, ensuring seamless integration with data scientists' workflows.Collaborate with...

  • Chief Data Pipeline Architect

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    Data Engineering RoleThis is an exciting opportunity for a Data Engineer to join our organization. As a key member of the team, you will be responsible for designing, developing, and maintaining data warehouses and data pipelines using Snowflake DB.The ideal candidate will have a strong background in data engineering and management, with experience in...

  • Senior Data Pipeline Specialist

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,10,000

    About Our OpportunityWe are looking for a talented Data Engineer to join our team and play a key role in designing, building, and maintaining scalable data pipelines and infrastructure.You will collaborate closely with developers, architects, analysts, and data scientists to ensure smooth, secure, and efficient data delivery across multiple products and...

  • Data Engineer

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 22,50,000 - ₹ 26,25,000

    Job OverviewWe are seeking a skilled professional to design, develop, and maintain scalable data platforms.This is a key role in shaping the future of cybersecurity by leveraging best practices for data engineering, platform development, and DevOps.Your ResponsibilitiesYou will be responsible for designing and developing data pipelines to extract, transform,...

  • Senior Data Engineer

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering LeadWe are seeking a highly skilled professional to lead our data engineering team. As a Senior ETL Developer, you will be responsible for designing and implementing scalable data pipelines using Databricks.Main Responsibilities:Collaborate with stakeholders to understand data requirements and translate them into technical...

  • Senior Data Engineer

    2 weeks ago

    Allahabad, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Azure Databricks engineers play a pivotal role in designing and implementing data processing systems leveraging cutting-edge big data technologies like Apache Spark. Key responsibilities encompass architecting scalable data pipelines, developing sophisticated data models, and optimizing database performance for seamless query execution.Key...