Product Data Ingestion Specialist
3 days ago
A senior web scraping engineer will be responsible for designing and implementing a high-throughput product data ingestion pipeline across hundreds of domains.
- Key Responsibilities:
- Design an HTTP-first crawler with Playwright fallback for JS-heavy pages.
- Implement sitemap diffing and conditional GETs (ETag/Last-Modified) for incremental runs.
- Build a lightweight 'needs JS?' classifier to auto-route HTTP vs Playwright.
- Enforce per-domain throttles/backoff and add URL normalization/canonicalization.
- Handle PDF discovery & download, apply Playwright browser automation resource budgets, and integrate third-party APIs as first-class sources.
- Owning automation & orchestration for scheduled runs, idempotent retries, and alerting.
- Create per-domain selectors with verification on hold-outs and ship observability metrics.
- 4+ years of Python experience, including 2+ years building production web crawlers at scale.
- Strong with Scrapy or aiohttp/asyncio and Playwright in production.
- Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
- Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching.
- Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
-
Cloud Data Ingestion Specialist
2 weeks ago
Bareilly, Uttar Pradesh, India beBeeDataIngestion Full time ₹ 15,00,000 - ₹ 20,00,000Cloud Data Ingestion SpecialistWe are seeking a highly skilled Cloud Data Ingestion Specialist to design, develop and optimize data ingestion pipelines for integrating multiple sources into Databricks.The ideal candidate will have expertise in CI/CD and Kubernetes, with hands-on experience in data pipelines, automation and infrastructure management. They...
-
IOT Data Solutions Specialist
4 days ago
Bareilly, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000IoT Data Solutions ArchitectWe are seeking a skilled specialist with 5 to 7 years of experience in designing and implementing scalable IoT data solutions.Develop and maintain large-scale IoT data ingestion, processing, and storage architectures.Utilize Python programming to build, optimize, and automate data pipelines leveraging AWS IoT services.Work...
-
Data Pipeline Engineer
20 hours ago
Bareilly, Uttar Pradesh, India beBeeIngestion Full time ₹ 15,00,000 - ₹ 25,00,000Job Opportunity: Ingestion SpecialistWe are seeking a skilled professional to develop and optimize data ingestion pipelines.The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support efficient ingestion workflows.Key Responsibilities:Design, develop, and maintain data ingestion pipelines for...
-
Senior Data Infrastructure Specialist
2 weeks ago
Bareilly, Uttar Pradesh, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,25,000Job DescriptionAbout the RoleWe are seeking a skilled professional to design and build scalable data pipelines that support real-time clinical workflows.This is a critical role where you will create robust systems to ingest, track, and update clinical events in real-time.You will work closely with software engineers, ML specialists, and our clinical partners...
-
Big Data Specialist
7 days ago
Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000Job Opportunity: Big Data SpecialistWe are seeking a highly skilled big data specialist to design, build, and maintain robust data pipelines for processing channel activity data from multiple sources.The ideal candidate will possess strong knowledge of cloud computing, programming languages like Python, and experience in handling data from various sources...
-
Data Migration Specialist
3 days ago
Bareilly, Uttar Pradesh, India beBeeDataMigration Full time ₹ 1,20,00,000 - ₹ 1,62,00,000Technical Data Migration SpecialistWe're seeking a seasoned technical expert to join our team as a Data Migration Specialist. In this role, you will be responsible for executing data migrations from legacy connectors into Snowflake/Nexla platforms.Key Responsibilities:Migrate large datasets from Fivetran and other legacy connectors into Snowflake/Nexla...
-
Big Data Specialist
5 days ago
Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 1,20,00,000 - ₹ 1,50,00,000Big Data Specialist Job DescriptionThis is a full-time role for an experienced specialist in data processing with one of our prominent clients.The specialist will be responsible for designing, deploying, and managing large-scale data processes to integrate information from various sources. Daily tasks include using ETL tools, performing data modeling,...
-
Cloud Integration Specialist
6 days ago
Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000Cloud Integration SpecialistWe are seeking a highly skilled Cloud Integration Specialist to join our team. As a Cloud Integration Specialist, you will be responsible for designing, building, and maintaining scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake.Data Pipeline Development & IntegrationDesign, build, and...
-
Data Engineering Specialist
5 days ago
Bareilly, Uttar Pradesh, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000Unlock Data Engineering OpportunitiesThis role is designed for individuals who excel in architecting and developing scalable data engineering solutions using Azure Databricks. Your primary responsibility will be to design, build, test, and deploy data pipelines that enable business insights.Data Ingestion and Processing:You will design and implement data...
-
Senior Data Architecture Specialist
3 days ago
Bareilly, Uttar Pradesh, India beBeeDataArchitecture Full time ₹ 14,05,000 - ₹ 20,10,000About the JobJob Title: Senior Data Architecture Specialist.We are seeking a highly skilled and experienced Data Architect to lead our data strategy, design, and implementation. The ideal candidate will have a deep understanding of data architecture principles, Snowflake, and Matillion.The successful candidate will be responsible for designing and...
