Senior Data Engineering Specialist

3 days ago

Salem, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 2,00,00,000
Senior Backend Data Engineer

This role involves designing and building data pipelines, connecting external systems and interfaces, and ensuring seamless integration of various tools and technologies.

Key Responsibilities:

  • Development of crawling/fetch pipelines using API-first approach with playwright/requests where permitted.
  • Parsing/normalization of job postings & CVs, deduplication/delta logic with seen hash, repost heuristics.
  • Embeddings/similarity search using Azure OpenAI and vector persistence in pgvector.
  • Integrations with HR4YOU (API/webhooks/CSV import), SerpAPI, BA job board, email/SMTP.
  • Batch/stream processing using Azure Functions/container jobs, retry/backoff, dead-letter queues.
  • Telemetry for data quality including freshness, duplicate rate, coverage, cost per 1,000 items.
  • Collaboration with FE for exports (CSV/Excel, presigned URLs) and admin configuration.

Requirements:

  • 4+ years of backend/data engineering experience.
  • Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), solid TypeScript for smaller services/SDKs.
  • Azure: Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics.
  • Messaging: Service Bus/Queues, idempotence & exactly-once semantics, pragmatic approach.
  • Databases: PostgreSQL, pgvector, query design & performance tuning.
  • Clean ETL/ELT patterns, testability (pytest), observability (OpenTelemetry).

Nice-to-haves:

  • NLP/IE experience (spaCy/regex/rapidfuzz), document parsing (pdfminer/textract).
  • Experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies (legally compliant).
  • Working method: API-first, clean code, trunk-based development, mandatory code reviews.
  • Tools/stack: GitHub, GitHub Actions/Azure DevOps, Docker, pnpm/Turborepo (Monorepo), Jira/Linear, Notion/Confluence.

  • Senior Data Engineering Specialist

    1 week ago

    Salem, Tamil Nadu, India beBeeEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data Engineering Specialist   We are seeking an experienced data engineering specialist to join our team.   Key Responsibilities:Design and develop end-to-end data integration solutions using Informatica IICS.Integrate IICS solutions with cloud-based data platforms such as Google Cloud Platform (GCP) and Google BigQuery (GBQ).Optimize...

  • Data Engineering Specialist

    7 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 35,00,000

    Job Overview:About the RoleWe are seeking a skilled Data Engineering Specialist to lead our data architecture efforts and design scalable data solutions.Key Responsibilities:Develop and maintain data engineering standards, guidelines, and best practices.Collaborate with cross-functional teams to implement data warehousing, big data, and data analytics...

  • Data Engineering Specialist

    7 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    About This RoleAs a Data Engineering Specialist, you will design, develop, and maintain large-scale data systems to process and analyze massive datasets. You will work with distributed data processing and storage technologies like Apache Spark, Hadoop, and cloud platforms such as AWS/GCP/Azure.This role requires expertise in developing efficient ETL...

  • Senior Data Engineer

    3 days ago

    Salem, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Engineer Job OpportunityWe are seeking a highly skilled Senior Data Engineer to join our technology team.The ideal candidate will lead the development of self-service data platforms for reporting and analytics, utilizing Databricks, SQL, Python, Spark, and Delta Lake on Azure.This role involves building scalable data solutions, ensuring best...

  • Data Engineering Specialist

    4 days ago

    Salem, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    ETL Data Specialist Job DescriptionThis role is an opportunity to work on designing, deploying, and managing ETL processes that integrate data from various sources.The ideal candidate will be responsible for creating ETL pipelines using Informatica and other tools to process and transform large datasets. Key responsibilities include:Designing, developing,...

  • Senior Data Scraping Specialist

    2 weeks ago

    Salem, Tamil Nadu, India beBeeScraping Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title: Senior Data Scraping SpecialistWe are seeking an accomplished and skilled professional to lead our data scraping initiatives.Key Responsibilities:Design and develop scalable data extraction solutions using tools like Scrapy and Python libraries.Mentor and guide a team of developers, overseeing project timelines and deliverables.Implement advanced...

  • Senior Data Engineer

    4 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    We are seeking a seasoned professional to contribute to the development of innovative analytical solutions used to improve drug, vaccine, and device safety monitoring and public health surveillance.Key Responsibilities:Design and architect future-state data driven products and servicesDevelop and optimize data lakehouse, Data Access APIs, and reporting...

  • Senior Data Engineer

    2 weeks ago

    Salem, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job OpportunityWe are seeking a highly skilled Senior Data Engineer to join our team and play a key role in scaling our analytics and reporting capabilities.This role is perfect for someone with deep technical expertise in ETL pipeline design, data modeling, and data infrastructure who thrives in a fast-paced, engineering-driven environment.You'll work...

  • Senior Data Engineering Professional

    5 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineering Specialist Job DescriptionWe are seeking a seasoned data engineering professional to lead the design, development, and deployment of scalable data pipelines and ETL processes using Databricks.Key Responsibilities:Collaborate with stakeholders to understand business requirements and translate them into technical solutions.Design and develop...

  • Senior Data Engineering Expert

    5 days ago

    Salem, Tamil Nadu, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Data Engineering OpportunitiesAs a Senior Data Engineer, you will play a key role in designing and implementing data models, ETL processes, and data warehouses using Snowflake.Design and implement data models and ETL processes to support business requirements.Develop and maintain large-scale data warehouses using Snowflake.Collaborate with cross-functional...