Senior Web Data Ingestion Specialist

1 week ago

Pushkar, Rajasthan, India beBeeDataIngestion Full time ₹ 1,50,00,000 - ₹ 2,00,00,000
Job Opening for High-Throughput Data Ingestion

We are building a high-throughput product data ingestion pipeline across hundreds of domains. You will own the crawling/extraction layer end-to-end: HTTP-first crawling with a Playwright fallback, per-domain learned selectors, and reliable PDF handling (datasheets/specs).

Key Responsibilities:
  • We need you to design an HTTP-first crawler with Scrapy or aiohttp and a Playwright fallback for JS-heavy pages.
  • You will implement sitemap diffing and conditional GETs for incremental runs.
  • The task involves building a lightweight classifier to auto-route HTTP vs Playwright.
  • You must enforce per-domain throttles and backoff.
  • You are responsible for adding URL normalization/canonicalization and de-duplication.
  • Handle PDF discovery and download with deduplication.
  • Apply Playwright browser automation resource budgets.
  • You will integrate third-party APIs as first-class sources.
  • You own automation and orchestration for scheduled runs and idempotent retries.
  • Ship observability metrics for per-site field coverage and error rates.
  • Maintain allow/deny paths and adhere to robots.txt and Terms of Service.
  • Containerize workers and collaborate on schemas/normalization.
Requirements:
  • A minimum of 4 years of Python experience, including 2+ years of building production web crawlers at scale.
  • Strong expertise in Scrapy or aiohttp and Playwright in production.
  • Practical proxy management and polite anti-bot tactics.
  • Confident with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Clear communication and strong ownership.

  • Data Ingestion Expert

    2 weeks ago

    Pushkar, Rajasthan, India beBeeDataIngestion Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job Title: Data Ingestion SpecialistAbout the Role:We are seeking an experienced professional to lead our data ingestion efforts. As a Data Ingestion Specialist, you will be responsible for designing and developing data pipelines that integrate multiple sources into Databricks.Your primary focus will be on integrating various data sources, implementing CI/CD...

  • Chief Data Ingestion Specialist

    1 week ago

    Pushkar, Rajasthan, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Data Engineer Job DescriptionA Cloud Data Engineer designs and develops data ingestion pipelines on Google Cloud Platform. They work closely with data scientists to support AI/ML workflows.This role involves architecting and managing scalable BigQuery data warehouses.The engineer will also be responsible for maintaining data quality, reliability, and...

  • Senior IoT Data Solutions Professional

    1 week ago

    Pushkar, Rajasthan, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    IoT Data Solutions EngineerWe are seeking a specialist with 5 to 7 years of experience in designing and implementing scalable IoT data solutions.Responsibilities include developing and maintaining data engineering solutions leveraging AWS IoT services, utilizing Python programming to build, optimize, and automate data pipelines.The ideal candidate should...

  • Senior Data Engineer

    2 weeks ago

    Pushkar, Rajasthan, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    As a key member of our data engineering team, the Senior Data Engineer - SSIS/SSRS Specialist will play a critical role in driving strategic direction and optimizing operations for our dynamic team.About the Role:This position is responsible for leading user inquiries, troubleshooting, and issue resolution for seamless experiences.The successful candidate...

  • Data Infrastructure Specialist

    2 weeks ago

    Pushkar, Rajasthan, India beBeeInfrastructure Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Job Title: Data Infrastructure SpecialistRole Summary:We seek an experienced data infrastructure specialist to craft, implement and optimize complex data solutions in a cloud-based environment.Design sophisticated data pipelines for ingestion and egress using Azure services.Build scalable data architectures using Databricks and Apache Spark.Troubleshoot data...

  • Data Infrastructure Specialist

    2 weeks ago

    Pushkar, Rajasthan, India beBeeDataInfrastructure Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Unlock your potential as a Data Infrastructure SpecialistWe're seeking a skilled professional to design and build scalable, fault-tolerant data infrastructure for large-scale applications.Key ResponsibilitiesCreate high-throughput systems for data ingestion, processing, and transformationDevelop synthetic datasets using state-of-the-art solutionsCollaborate...

  • Cloud Data Solutions Specialist

    1 week ago

    Pushkar, Rajasthan, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Cloud Data Solutions SpecialistAbout the Role:We are seeking an experienced data professional to join our team.The ideal candidate will have 7+ years of experience in designing, building, and maintaining large-scale data pipelines on Azure.You will be responsible for ensuring the smooth operation of our data infrastructure, including data...

  • Web Data Extraction Specialist

    2 weeks ago

    Pushkar, Rajasthan, India beBeeDataExtractor Full time US$ 80,000 - US$ 1,00,000

    Transforming unstructured web data into actionable insights requires designing and maintaining efficient web crawlers that extract valuable information from the vast internet landscape.Key ResponsibilitiesWe are seeking a skilled Web Data Extractor to develop and refine crawlers using Python-based tools and frameworks, ensuring maximum efficiency and...

  • Senior Data Extraction Engineer

    2 weeks ago

    Pushkar, Rajasthan, India beBeeDataExtraction Full time ₹ 12,00,000 - ₹ 20,00,000

    Web Scraping & OCR SpecialistAs a seasoned Web Scraping & OCR Specialist, you will be responsible for crafting and refining data extraction solutions using Python.Design and develop Python scripts for web scraping from structured and unstructured sources.Implement Optical Character Recognition (OCR) solutions to extract text/data from scanned images, PDFs,...

  • Data Integration Specialist

    2 weeks ago

    Pushkar, Rajasthan, India beBeeDataIntegration Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Data Integration SpecialistThis is a full-time opportunity for an experienced Data Integration Specialist to join our team in India. The candidate will be responsible for designing, deploying, and managing data integration processes to integrate data from various sources.About the Role:Design, develop, and maintain data pipelines using Informatica...