Senior Data Engineering Specialist

3 days ago

Gandhinagar, Gujarat, India beBeeBackendEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

As a Senior Backend Data & Integration Engineer, you will be responsible for designing and implementing crawling/fetch pipelines using API-first approach with playwright/requests. You will also work on parsing/normalization of job postings & CVs, deduplication/delta logic, embeddings/similarity search, integrations with HR4YOU, SerpAPI, BA job board, email/SMTP, batch/stream processing, telemetry for data quality, collaboration with FE for exports and admin configuration.

Key Responsibilities:
  • Design and implement crawling/fetch pipelines using API-first approach
  • Parsing/normalization of job postings & CVs, deduplication/delta logic
  • Embeddings/similarity search using Azure OpenAI and vector persistence in pgvector
  • Integrations with HR4YOU, SerpAPI, BA job board, email/SMTP
  • Batch/stream processing using Azure Functions/container jobs, retry/backoff, dead-letter queues
  • Telemetry for data quality including freshness, duplicate rate, coverage, cost per 1,000 items
  • Collaboration with FE for exports (CSV/Excel) and admin configuration
Must Have Requirements:
  • 4+ years of backend/data engineering experience
  • Proficiency in Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), solid TypeScript for smaller services/SDKs
  • Azure: Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics
  • Messaging: Service Bus/Queues, idempotence & exactly-once semantics
  • Databases: PostgreSQL, pgvector, query design & performance tuning
  • Clean ETL/ELT patterns, testability (pytest), observability (OpenTelemetry)
Nice-to-have:
  • NLP/IE experience (spaCy/regex/rapidfuzz), document parsing (pdfminer/textract)
  • Experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies
  • Working method: API-first, clean code, trunk-based development, mandatory code reviews
  • Tools/stack: GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, Notion/Confluence

  • Data Engineering Specialist

    2 weeks ago

    Gandhinagar, Gujarat, India beBeeDataEngineering Full time ₹ 10,00,000 - ₹ 14,00,000

    Data Engineering SpecialistWe are seeking an experienced Data Engineer to join our global engineering team. This is a unique opportunity to bring data engineering expertise to a multi-year project initiative.Key Responsibilities:Design and Implement Scalable Data PipelinesDevelop and Maintain Efficient Data Processing SystemsCollaborate with Cross-Functional...

  • Senior Data Engineer

    2 weeks ago

    Gandhinagar, Gujarat, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job TitleA senior data engineer is required to develop and optimize ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Synapse Analytics.This role involves designing and implementing scalable data architectures following Medallion Architecture and best practices.The ideal candidate will have experience in Azure Data Engineering with Data...

  • Senior Data Engineering Specialist

    4 days ago

    Gandhinagar, Gujarat, India beBeeSenior Full time ₹ 20,00,000 - ₹ 30,00,000

    We are seeking an experienced Senior ETL Developer to join our organization. The ideal candidate will have a proven track record of designing, developing, and deploying scalable data pipelines and ETL processes using Databricks.Key Responsibilities:Design and develop efficient ETL processes and data pipelines that meet business requirementsOptimize Spark...

  • Lead Data Engineering Specialist

    7 days ago

    Gandhinagar, Gujarat, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Opportunity: Senior Data EngineerWe are seeking an experienced Data Engineer to join our team. As a Senior Data Engineer, you will be responsible for designing and implementing large-scale data systems.Main Responsibilities:Design and implement scalable data pipelines using Azure and AWS Databricks.Develop and maintain high-quality data processing...

  • Senior Data Analytics Specialist

    2 weeks ago

    Gandhinagar, Gujarat, India beBeeDataAnalytics Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Job Title: Senior Data Analytics SpecialistIn this role, you will play a critical part in the migration of enterprise data solutions from SAP HANA to Snowflake. This involves utilizing expertise in data analytics platforms, particularly SAP HANA and Snowflake.Key Responsibilities:Collaborate with business stakeholders to understand and identify the analytics...

  • IoT Data Engineer Specialist

    4 days ago

    Gandhinagar, Gujarat, India beBeeIoT Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: IoT Data Engineer Specialist">We are seeking a highly skilled IoT Data Engineer to join our team.Key Responsibilities:Develop and maintain data engineering solutions leveraging AWS IoT services, utilizing Python programming to build, optimize, and automate data pipelines.Work with large-scale IoT data ingestion, processing, and storage...

  • Lead Data Engineering Specialist

    4 days ago

    Gandhinagar, Gujarat, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Senior Data Engineer Job DescriptionWe are seeking a highly skilled Senior Data Engineer to lead our data engineering efforts. The ideal candidate will have expertise in cloud platforms, ETL processes, data warehousing, and big data technologies.Job RequirementsTo design, deploy, and maintain large-scale data platforms with high efficiency and scalability.To...

  • Senior Clinical Data Specialist

    1 week ago

    Gandhinagar, Gujarat, India beBeeClinical Full time ₹ 1,04,000 - ₹ 1,30,878

    Job Title: Senior Clinical Data SpecialistAbout the RoleThis is an exciting opportunity for a detail-oriented and analytical individual to join our team as a Senior Clinical Data Specialist. In this role, you will be responsible for assessing and handling clinical data and case reports, particularly Suspected Unexpected Serious Adverse Reactions (SUSAR).You...

  • Senior Data Engineer Position

    4 days ago

    Gandhinagar, Gujarat, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Description:As a senior data engineer, you will be responsible for designing and developing large-scale data pipelines using Azure Data Factory and Databricks. This role involves collaborating with cross-functional teams to identify business requirements and develop data-driven solutions.The ideal candidate will have a strong understanding of data...

  • Expert Data Integration Specialist

    5 days ago

    Gandhinagar, Gujarat, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Senior Data Integration SpecialistWe are seeking a seasoned Senior Data Integration Specialist to lead our Informatica to SSIS migration projects. Key Responsibilities:Project Lead: Direct and execute Informatica to SSIS migration projects, ensuring accuracy, scalability, and performance.Design and develop robust ETL processes for data integration and...