High-Throughput Web Data Engineer

4 days ago

Bhavnagar, Gujarat, India beBeeScraping Full time ₹ 1,50,00,000 - ₹ 2,00,00,000
Senior Web Scraping Engineer Job Description

We are seeking a highly skilled Senior Web Scraping Engineer to build a high-throughput product data ingestion pipeline across hundreds of domains. This is an end-to-end role, encompassing crawling and extraction, including HTTP-first crawling with a Playwright fallback.

Key Responsibilities:
  • Design an HTTP-first crawler with Scrapy or aiohttp, supplemented by a Playwright fallback for complex web pages.
  • Implement sitemap diffing and conditional GETs (ETag/Last-Modified) to ensure incremental runs.
  • Develop a lightweight classifier to determine whether JavaScript-heavy pages require the use of Playwright.
  • Implement per-domain throttles and backoff strategies to prevent overload and 429/503 errors.
  • Apply URL normalization and de-duplication techniques to eliminate duplicates.
  • Integrate third-party APIs as first-class sources, handling authentication, pagination, and rate limits.
  • Own automation and orchestration for scheduled runs, idempotent retries, and alerting.
  • Develop per-domain selectors using YAML and verify them on hold-out sets.
  • Monitor observability metrics, such as field coverage, error rates, retries, and average page time.
  • Maintain allow/deny paths and adhere to robots.txt and Terms of Service.
Requirements:
  • 4+ years of experience in Python development, with at least 2 years spent building production web crawlers at scale.
  • Strong expertise in Scrapy, aiohttp, or asyncio, and Playwright or Puppeteer.
  • Familiarity with proxy management, anti-bot tactics, and per-domain rate limiting.
  • Practical experience with ETag, Last-Modified, retries, backoff, and HTTP caching.
  • Confidence in CSS/XPath, schema.org, and HTML parsing.
  • API knowledge: consuming REST/GraphQL and building internal services.
  • Automation/orchestration skills: scheduling, monitoring, and queues (Redis/Kafka).
  • Experience with PDF handling, file integrity checks, and Linux basics.
  • Clear communication and strong ownership.

  • High-Performance Data Architect

    6 days ago

    Bhavnagar, Gujarat, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Engineering RoleWe're seeking an experienced Data Engineer to join our team responsible for building and scaling data infrastructure. This role requires expertise in large-scale data processing for AI model development.About the Position:You'll work directly with researchers to accelerate experiments, develop new datasets, improve infrastructure...

  • Lead Frontend Web Engineer

    7 days ago

    Bhavnagar, Gujarat, India beBeeFrontend Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Overview:We are seeking a seasoned web engineer to join our development team and contribute to the creation of cutting-edge web applications. The ideal candidate will have a strong background in frontend development, with expertise in modern tooling and web standards.Key Responsibilities:Design and develop scalable, maintainable frontend features that...

  • Web Software Engineer

    5 days ago

    Bhavnagar, Gujarat, India beBeeSoftwareDevelopment Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Senior Web Application DeveloperWe are looking for a highly skilled and experienced Senior Web Application Developer to join our team. This is a key role in the development of scalable, maintainable, and high-performance web applications that handle large amounts of data.Key Responsibilities:To design, build, test, and deploy web applications using modern...

  • High-Performance Data Ingestion Specialist

    2 days ago

    Bhavnagar, Gujarat, India beBeeWebScraping Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    We're building a high-throughput data ingestion pipeline across hundreds of domains. As a Senior Web Scraping Engineer, you'll own the crawling/extraction layer end-to-end: HTTP-first crawling with a Playwright fallback, per-domain learned selectors, and reliable PDF handling (datasheets/specs).Key ResponsibilitiesDesign an HTTP-first crawler (Scrapy or...

  • Senior Design Engineer for High-Speed Interconnect Architectures

    2 weeks ago

    Bhavnagar, Gujarat, India beBeeEthernet Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Senior Design Engineer for Ethernet Switch ArchitecturesOverview:We are seeking a seasoned IP/RTL Design Engineer with over 10 years of experience to design and develop intellectual property (IP) and register-transfer level (RTL) for Ethernet switches tailored for AI datacenter backend networks.Design and optimize IP blocks (MAC, PCS, packet...

  • Senior Cloud Development Engineer

    3 days ago

    Bhavnagar, Gujarat, India beBeeDevelopment Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Cloud Architect PositionWe are seeking a highly skilled Software Architect with deep expertise in AWS, serverless architectures, and TypeScript to join our team.About the Role:Rapidly familiarize yourself with an ongoing project and its architecture.Independently design, develop, and deploy new features using AWS serverless technologies.Define and implement...

  • Chief Data Engineering Specialist

    7 days ago

    Bhavnagar, Gujarat, India beBeeDataScientist Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: **Lead Data Scientist/Engineer**: We are seeking a highly skilled and experienced professional to lead our data science team. The successful candidate will have a strong background in data engineering, Python programming, and web scraping.">Develop scalable data solutions using Scrapy and relevant libraries.Manage project timelines and...

  • Senior Web Developer Position

    4 days ago

    Bhavnagar, Gujarat, India beBeeFrontendDeveloper Full time ₹ 1,50,00,000 - ₹ 2,01,00,000

    About UsWe are seeking a talented Web Engineer to join our team. You will play a key role in building and optimizing high-performance, scalable, and intuitive web applications that reach millions of users daily.The ideal candidate will have strong expertise in modern web frameworks and tools, including JavaScript, TypeScript, React.js, and state management...

  • Data Architect

    2 weeks ago

    Bhavnagar, Gujarat, India beBeeDataEngineer Full time US$ 2,00,000 - US$ 2,50,000

    About us:We are a healthcare technology company that aims to make nutritious food accessible and affordable for everyone.We have a robust network of Registered Dietitians who provide personalized nutrition counseling and guidance to our members. Our platform integrates dietary assessments, nutrition counseling, online food ordering, and cost-effective meal...

  • Senior Web Engineer

    5 days ago

    Bhavnagar, Gujarat, India beBeeSoftware Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Description:We are seeking an experienced software engineer to lead the design and implementation of robust web services.This is a challenging role for individuals with expertise in programming languages such as React, Redux, and GraphQL.