Web Crawling Engineer

6 days ago

Nagpur, Maharashtra, India beBeeScalable Full time ₹ 1,00,00,000 - ₹ 1,50,00,000
Job Overview

This role encompasses the design and implementation of large-scale web crawling and scraping systems. The ideal candidate will have expertise in building production-grade web crawlers using Python, with a strong focus on scalability, reliability, and performance.

  • Design an HTTP-first crawler utilizing Scrapy or aiohttp, with Playwright as a fallback for JavaScript-heavy pages.
  • Implement sitemap diffing and conditional GETs to enable incremental runs.
  • Develop a lightweight classifier to determine whether HTTP or Playwright is required.
  • Enforce per-domain throttling and backoff with concurrent limits.
  • Apply URL normalization, canonicalization, and deduplication techniques.
  • Handle PDF discovery and download, incorporating deduplication and size/concurrency caps.
  • Implement Playwright browser automation resource budgets.
  • Integrate third-party APIs as first-class sources, including authentication, pagination, and rate limiting.
  • Own automation and orchestration for scheduled runs, incorporating idempotent retries and alerting.
  • Create per-domain selectors with verification on hold-outs.
  • Ship observability metrics.

Maintain allow/deny paths and adhere to robots.txt and Terms of Service. Containerize workers and provide runbooks/CI.

Requirements
  • 4+ years of Python experience, including 2+ years of building production web crawlers at scale.
  • Strong experience with Scrapy, aiohttp, Playwright, and proxy management.
  • Confident with CSS/XPath, schema.org, JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery for scheduled runs and monitoring.
  • PDF handling and file integrity checks.
  • Queues, Docker, Linux basics; comfort with logs/metrics.
Nice to Have
  • Go or Node.js experience for high-performance crawlers.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics.
  • Workflow engines: Airflow/Temporal/Argo/Celery.
  • Document extraction: Textract/Tika/Camelot/Tabula.
  • Search/analytics: Elasticsearch/OpenSearch; warehousing (Snowflake/Postgres).

About This Role

This position requires a skilled professional who can develop and maintain large-scale web crawling and scraping systems. If you are passionate about building scalable and reliable software, this might be the perfect opportunity for you.

  • Senior Web Crawler Specialist

    1 week ago

    Nagpur, Maharashtra, India beBeeWebCrawler Full time ₹ 9,00,000 - ₹ 12,00,000

    We are seeking a professional with expertise in web crawling and data extraction to join our team.Key Responsibilities:Maintain and enhance existing web scraping and data crawling projects using Python-based tools and frameworks.Develop and refine crawlers for extracting valuable insights from the web.Utilize browser automation tools (e.g. Playwright,...

  • Senior AI Automation Specialist

    2 weeks ago

    Nagpur, Maharashtra, India beBeeAutomation Full time ₹ 50,000 - ₹ 70,000

    AI Automation DeveloperWe're looking for a skilled professional to design and build an intelligent web automation system.This role involves combining AI techniques with web crawling/automation tools to create a scalable and efficient solution.Key Responsibilities:Design and develop an AI-powered web automation system capable of interacting with websites...

  • Data Engineer

    4 days ago

    Nagpur, Maharashtra, India beBeeData Full time ₹ 9,00,000 - ₹ 12,00,000

    Data Engineer - Scalable Solutions SpecialistWe are seeking a highly skilled Data Engineer to join our team and help build scalable data solutions. As we develop our technology platform from scratch, you will have the opportunity to shape our technical vision and architecture.This role is ideal for someone who thrives on early-stage challenges, loves...

  • Web Software Engineer

    7 days ago

    Nagpur, Maharashtra, India beBeeSoftware Full time ₹ 20,00,000 - ₹ 25,00,000

    We are seeking a skilled Web Software Engineer to join our team. This role involves designing, building, testing and deploying scalable, reusable and maintainable web applications that can handle large amounts of data.Key Responsibilities:Design and build high-quality web applications using modern web development tools and technologies.Write new UI...

  • Senior Web Development Manager

    2 weeks ago

    Nagpur, Maharashtra, India beBeeTechLead Full time ₹ 14,99,999 - ₹ 19,99,999

    Job Title: Tech LeadAbout the RoleWe are seeking a highly skilled and experienced Tech Lead to join our team.The ideal candidate will have a strong understanding of web development, programming techniques, and tools.Over 120 high-traffic eCommerce websites serving customers around the worldBusiness analytics that determines product and catalog selection,...

  • Senior Web Application Engineer

    2 weeks ago

    Nagpur, Maharashtra, India beBeeSoftware Full time ₹ 25,00,000 - ₹ 35,00,000

    About Us: We are a leading technology solutions provider dedicated to delivering business growth and innovation. Our team of skilled professionals is passionate about engineering and creativity, empowering agile teams to execute ideas and solve complex challenges.We are seeking an experienced Web Application Developer to join our engineering squad. As a key...

  • Senior Web Software Engineer

    2 weeks ago

    Nagpur, Maharashtra, India beBeeBackend Full time ₹ 22,14,000 - ₹ 26,10,000

    Key Responsibilities:Serving as a core member of an agile team that drives user story analysis and elaboration designs and develops responsive web applications using the best engineering practices.Performing hands-on software development typically spending most of time actually writing code and unit tests doing proof of concepts conducting code reviews and...

  • Web Application Engineer

    2 weeks ago

    Nagpur, Maharashtra, India beBeeDeveloper Full time ₹ 18,00,000 - ₹ 20,00,000

    Full Stack Web Developer Job OpportunityThis position requires a skilled Full Stack Web Developer to design, develop, and ship high-performance web experiences in Next.js. The ideal candidate will have experience with headless CMSs, modern front-end frameworks, and DevOps tools.Key Responsibilities:Develop new features using Next.js and React, ensuring...

  • Full Stack Web Solutions Engineer

    2 weeks ago

    Nagpur, Maharashtra, India beBeeDeveloper Full time ₹ 15,00,000 - ₹ 25,00,000

    At Hexnode, we value diversity and believe in creating a sense of belonging for our employees.We recognize the rapid evolution of technology and society that impacts our industry, and prioritize equipping our employees with diverse opportunities and empowering them with a wide range of skills. Fueling the transformation to a seamless ecosystem of connected...

  • Web Development Lead

    1 week ago

    Nagpur, Maharashtra, India beBeeEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Opportunity: Senior EngineerWe are seeking a highly skilled Senior Engineer to lead our web development team.The ideal candidate will have extensive experience in developing and maintaining web applications using Python, Django, PHP, JavaScript, HTML, CSS, and Ajax.The successful candidate will be responsible for creating and managing REST APIs for...