
Data Pipeline Architect
5 days ago
We are seeking a highly skilled Senior Backend Data & Integration Engineer to join our team. The successful candidate will be responsible for designing and implementing data pipelines, connecting external systems and interfaces, and collaborating with the frontend team.
- Development of crawling/fetch pipelines using API-first approach with playwright/requests where permitted
- Parsing/normalization of job postings & CVs, deduplication/delta logic including seen hash, repost heuristics
- Embeddings/similarity search using Azure OpenAI and vector persistence in pgvector
- Integrations: HR4YOU (API/webhooks/CSV import), SerpAPI, BA job board, email/SMTP
- Batch/stream processing using Azure Functions/container jobs, retry/backoff, dead-letter queues
- Telemetry for data quality including freshness, duplicate rate, coverage, cost per 1,000 items
Key Qualifications:
- 4+ years of backend/data engineering experience
- Python (FastAPI, pydantic, httpx/requests, Playwright/Selenium), solid TypeScript for smaller services/SDKs
- Azure: Functions/Container Apps or AKS jobs, Storage/Blob, Key Vault, Monitor/Log Analytics
- Messaging: Service Bus/Queues, idempotence & exactly-once semantics, pragmatic approach
- Databases: PostgreSQL, pgvector, query design & performance tuning
Nice-to-Have Qualifications:
- NLP/IE experience (spaCy/regex/rapidfuzz), document parsing (pdfminer/textract)
- Experience with license/ToS-compliant data retrieval, captcha/anti-bot strategies (legally compliant)
- Working method: API-first, clean code, trunk-based development, mandatory code reviews
-
Data Pipeline Specialist
6 days ago
Erode, Tamil Nadu, India beBeeDataEngineering Full time ₹ 10,00,000 - ₹ 20,00,000Job Description:Seeking a data engineering professional to build scalable data pipelines and cloud-native data solutions on GCP. Ideal candidate will be proficient in Pub/Sub, Dataflow, Cloud Storage, and BigQuery, with a foundational understanding of AI/ML workflows using Vertex AI.Key Responsibilities:Design and develop robust data ingestion pipelines...
-
Chief Data Pipeline Architect
1 week ago
Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000About the RoleWe are seeking a Senior Data Engineer to design, build, and operate scalable data pipelines.This is a critical role where you will create scalable pipelines that process clinical encounter data, retrieve associated medical documents, and deliver validated information to machine learning systems for inference.Your ResponsibilitiesDesign and...
-
Chief Data Pipeline Architect
2 weeks ago
Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 30,00,000Key Roles and ResponsibilitiesWe are seeking a highly skilled professional to join our data engineering team. As a key member of the team, you will play a vital role in designing and implementing efficient data pipelines using modern DevOps tools.Develop and manage continuous integration and delivery pipelines using GitHub Actions or equivalent...
-
Chief Data Pipeline Architect
7 days ago
Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000Job Title: Senior Data EngineerWe are seeking an exceptional Azure Data Engineer with 7+ years of experience to design, develop, and optimize data pipelines and integration solutions in a cloud environment.Key Responsibilities:Design and implement advanced ETL pipelines for batch data ingestion and egress.Develop scalable data solutions using Azure Data...
-
Principal Data Pipeline Architect
2 weeks ago
Erode, Tamil Nadu, India beBeeDataEngineering Full time ₹ 10,00,000 - ₹ 15,00,000Job Title: Data EngineerDescription:We are seeking an experienced Data Engineer to take on the challenge of building and optimizing data pipelines. The successful candidate will ensure that data is clean, secure, and accessible for analytics and operations.Responsibilities:Design and optimize scalable data pipelines and architectures using cutting-edge...
-
Data Architect Lead
7 days ago
Erode, Tamil Nadu, India beBeeData Full time ₹ 1,20,00,000 - ₹ 1,50,00,000Job Title: Data ArchitectWe are seeking a seasoned professional to lead our data-driven projects as a Data Architect. This role involves designing and developing features in our existing Data Warehouse, establishing connections between Engineering, Product, and Analytics/Data Science teams, implementing, updating, and maintaining batch ETL pipelines,...
-
Chief Data Pipeline Architect
1 week ago
Erode, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000About the RoleThis is a technical professional role that involves designing and building scalable data pipelines using Azure technologies.Key Responsibilities:We are looking for an experienced data engineer to collaborate with our analytics and business teams to improve data models and increase data accessibility.The successful candidate will be responsible...
-
Seeking Experienced Data Architect
7 days ago
Erode, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000Job Title: Data ArchitectWe seek a seasoned Data Architect to design and implement scalable, secure data pipelines on Google Cloud Platform. The ideal candidate will have expertise in data engineering with 5+ years of experience in designing, building, and optimizing complex data architectures.
-
Data Pipeline Specialist
6 days ago
Erode, Tamil Nadu, India beBeeDataPipelineSpecialist Full time ₹ 12,00,000 - ₹ 21,50,000Job Title: Data Pipeline SpecialistAre you a skilled engineer with expertise in data pipeline architecture and big data technologies? Do you have hands-on experience with Spark, DBT, and cloud services?We are looking for a talented individual to join our team as a Data Pipeline Specialist.Create and maintain optimal data pipeline architecturesDesign and...
-
Data Architect
2 weeks ago
Erode, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000We are seeking a skilled Data Architect to develop, optimize and maintain data pipelines that drive business growth and analytics initiatives. This role will focus on building robust workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration primarily using...