High-Performance Web Crawler Developer

5 days ago

India beBeeSpecialist Full time ₹ 1,50,00,000 - ₹ 2,00,00,000
Senior Web Scraping Engineer

We are seeking an experienced Web Scraping Specialist to join our team. The successful candidate will own the crawling/extraction layer end-to-end, designing and implementing a scalable solution that meets our needs.

The ideal candidate will have strong expertise in Scrapy or aiohttp and Playwright, with practical experience in proxy management, anti-bot tactics, and rate limiting. They will also be confident in CSS/XPath, schema.org/JSON-LD, and HTML parsing.

Key Responsibilities:
  • Design and implement a scalable web crawler using HTTP-first and Playwright fallbacks.
  • Develop a lightweight 'needs JS?' classifier to determine when JavaScript is required for crawling.
  • Implement sitemap diffing and conditional GETs to optimize crawling efficiency.
  • Enforce per-domain throttles and backoff strategies to prevent overwhelming server resources.
  • Handle PDF discovery and download, integrating third-party APIs as needed.
Requirements:
  • 4+ years of Python experience, including 2+ years building production web crawlers at scale.
  • Strong expertise in Scrapy or aiohttp and Playwright, with practical experience in proxy management and anti-bot tactics.
  • Confident in CSS/XPath, schema.org/JSON-LD, and HTML parsing.
Benefits:
  • Opportunity to work on a high-profile project with significant impact.
  • Collaborative and dynamic work environment.
  • Professional growth and development opportunities.
How We Work:
  • We prioritize simple designs that are easy to operate at scale.
  • We track coverage and freshness as north-star metrics.
  • We ship in small, measurable increments.

  • High-Tech Web Crawler Specialist

    7 days ago

    India beBeeEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Description:">We're looking for a seasoned web scraping engineer to join our team. Your primary responsibility will be designing and implementing a high-throughput product data ingestion pipeline that can handle hundreds of domains.">As a web scraping engineer, you'll be responsible for the crawling/extraction layer end-to-end, including HTTP-first...

  • High-Performance Data Extractor

    5 days ago

    India beBeeWeb Full time ₹ 6,00,000 - ₹ 8,00,000

    Job DescriptionWe are seeking a skilled Web Scraping Engineer to join our team. This is an exciting opportunity to build a high-throughput product data ingestion pipeline across hundreds of domains.As a Senior Web Scraping Engineer, you will own the crawling/extraction layer end-to-end: crawling with a fallback, per-domain learned selectors, and reliable PDF...

  • Master Web Data Extractor

    2 weeks ago

    India beBeeData Full time ₹ 7,50,000 - ₹ 15,00,000

    Web Crawling Engineer PositionWe are seeking a skilled Web Crawling Engineer who will be responsible for designing and implementing web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling...

  • Senior Python Web Engineer

    4 days ago

    India TripleChoice Inc Full time

    Senior Web Scraping Engineer (Python) — India (Remote) Employment type: Full-time (open to contract-to-hire) Work location: Remote in India Prefer 2–3 hours/day with Pacific Time (PST) About the role We’re building a high-throughput product data ingestion pipeline across hundreds of domains. You’ll also drive the automation around scheduling,...

  • Senior Python Web Engineer

    5 days ago

    India TripleChoice Inc Full time

    Senior Web Scraping Engineer (Python) — India (Remote) Employment type: Full-time (open to contract-to-hire) Work location: Remote in India Prefer 2–3 hours/day with Pacific Time (PST)About the role We're building a high-throughput product data ingestion pipeline across hundreds of domains. You'll also drive the automation around scheduling,...

  • Junior Web Crawling Engineer

    2 weeks ago

    India Forage AI Full time

    Job DescriptionWe are seeking a Junior Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping...

  • Lead Data Pipeline Developer

    5 days ago

    India beBeeData Full time ₹ 6,00,000 - ₹ 8,00,000

    Senior Web Engineer Job DescriptionWe are seeking an experienced Senior Web Engineer to build a high-throughput product data ingestion pipeline across hundreds of domains. The successful candidate will design, develop, and implement scalable web crawlers using Python.The role involves collaborating with the data team to integrate vendor/public APIs...

  • Senior Product Data Ingestion Specialist

    7 days ago

    India beBeeCrawler Full time ₹ 15,00,000 - ₹ 28,00,000

    High-Performance Web Crawling Engineer">We are looking for an experienced engineer to build a high-throughput product data ingestion pipeline across hundreds of domains. The successful candidate will own the crawling/extraction layer end-to-end, designing and implementing an HTTP-first crawler with a Playwright fallback for JavaScript-heavy pages.">The role...

  • Technical Lead for High-Performance Web Development

    1 week ago

    India beBeeCareer Full time ₹ 15,00,000 - ₹ 25,00,000

    Technical Lead for Scalable Web ApplicationsAbout Position:This is a challenging role for a seasoned technical architect to lead the development of complex, scalable web applications that address business problems in travel. The ideal candidate will have expertise in Python (Django and Flask) and React frontend technologies.Role: Technical LeadLocation:...

  • Senior Data Extraction Specialist

    2 weeks ago

    India beBeeData Full time ₹ 15,00,000 - ₹ 28,00,000

    Web Crawling Engineer OpportunityWe are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong programming skills in Python and experience in web scraping frameworks, browser automation tools, and handling...