Senior Web Crawler Specialist

1 week ago

Alleppey, Kerala, India beBeeEngineer Full time US$ 12,00,000 - US$ 15,00,000
Web Scraping Engineer Role

We are seeking a skilled Web Scraping Engineer to develop high-throughput web crawling and scraping solutions.

This role involves designing and implementing web crawlers using Python, Scrapy, aiohttp, and Playwright.

  • Design an HTTP-first crawler with a Playwright fallback for JavaScript-heavy pages.
  • Implement sitemap diffing and conditional GETs for incremental runs.
  • Build a lightweight classifier to route HTTP and Playwright requests.
  • Enforce per-domain throttles and backoff, as well as URL normalization and canonicalization.
  • Handle PDF discovery and download with size and concurrency constraints.

The ideal candidate will have 4+ years of experience with Python and 2+ years of building production web crawlers at scale. They should be proficient in Scrapy or aiohttp/asyncio and Playwright in production, as well as confident in CSS/XPath, schema.org/JSON-LD, and HTML parsing.

They should also have hands-on experience with ETag/Last-Modified, retries, backoff, and HTTP caching, as well as API consumption and building small internal services.

Automation and orchestration experience is required, including Airflow/Celery for scheduled runs and monitoring. Queues, Docker, Linux basics, and comfort with logs/metrics are essential.

Strong communication skills and ownership are crucial for this role.

Key Requirements

Must-haves:

  • 4+ years of Python experience, including 2+ years building production web crawlers.
  • Experience with Scrapy or aiohttp/asyncio and Playwright in production.
  • Confidence in CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • Hands-on experience with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Experience with API consumption and building small internal services.
  • Automation and orchestration experience with Airflow/Celery.
  • Queues, Docker, Linux basics, and comfort with logs/metrics.

Nice-to-haves:

  • Go or Node.js experience for high-performance crawlers.
  • Cloud experience with AWS/GCP, S3, ECS/Kubernetes; Infrastructure as Code basics.
  • Workflow engine experience with Airflow/Celery/Argo.
  • Document extraction experience with Textract/Tika/Camelot/Tabula.
  • Search/analytics experience with Elasticsearch/OpenSearch; warehousing (Snowflake/Postgres).
Our Approach
  • Ship in small, measurable increments.
  • Track coverage and freshness as key metrics.
  • Prefer simple designs that are easy to operate at scale.

  • Senior Data Extraction Engineer

    2 weeks ago

    Alleppey, Kerala, India beBeeCrawling Full time ₹ 10,00,000 - ₹ 15,00,000

    Job OpportunityWe are seeking a Web Crawling Specialist who will be responsible for developing and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.The ideal candidate will have strong programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping...

  • Senior Web Scraping Engineer

    1 week ago

    Alleppey, Kerala, India TripleChoice Inc Full time

    Senior Web Scraping Engineer (Python) — India (Remote)Employment type: Full-time (open to contract-to-hire)Work location: Remote in IndiaTime overlap: Prefer 2–3 hours/day with Pacific Time (PST)About the roleWe're building a high-throughput product data ingestion pipeline across hundreds of domains. You'll own the crawling/extraction layer...

  • Web Interface Specialist

    1 week ago

    Alleppey, Kerala, India beBeeWebInterface Full time ₹ 12,00,000 - ₹ 24,00,000

    Job Title: Web Interface SpecialistWe are seeking a skilled web interface specialist to build and maintain the user experience of our applications.Bachelor's degree in computer science or related field, or equivalent experience4-6 years of professional experience in software developmentKey Responsibilities:Develop high-quality web interface code using...

  • Senior Web Developer

    2 weeks ago

    Alleppey, Kerala, India beBeeDeveloper Full time ₹ 50,00,000 - ₹ 70,00,000

    Web Design SpecialistWe are seeking an experienced Web Design Specialist to convert high-fidelity mockups into fast, responsive, and pixel-perfect websites.Your responsibilities will include:Converting Figma/Sketch/Adobe XD mockups into responsive, pixel-perfect websitesWriting clean, reusable HTML, CSS (Flexbox/Grid), JS (ES6+) + PHP...

  • Senior Web Content Specialist

    2 weeks ago

    Alleppey, Kerala, India beBeeContent Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior AEM Content AuthorWe are seeking a skilled content strategist to join our team. The ideal candidate will be responsible for creating, managing and maintaining high-quality web content using Adobe Experience Manager (AEM).Required Skills and QualificationsStrong understanding of AEM and its features.Ability to work with cross-functional...

  • Senior Software Quality Assurance Specialist

    1 week ago

    Alleppey, Kerala, India beBeequality Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Senior Software Quality Assurance SpecialistWe are seeking a skilled Senior Software Quality Assurance Specialist to join our team. This role will be responsible for ensuring the accuracy and reliability of our AI-driven medical analysis tools.The ideal candidate will have 3+ years of manual QA experience with web applications, including test case...

  • Enterprise Web Design Specialist

    6 days ago

    Alleppey, Kerala, India beBeeUIUX Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Enterprise Web Design SpecialistWe are seeking a talented Enterprise Web Design Specialist to lead the end-to-end design process for our web applications. This is an exciting opportunity for a passionate designer to collaborate with cross-functional teams and drive design solutions that meet business objectives and enhance user experience.The ideal candidate...

  • Senior Web Development Specialist

    1 week ago

    Alleppey, Kerala, India beBeeDevelopment Full time ₹ 12,00,000 - ₹ 15,00,000

    Web Application Developer Role">Job Title: Web Application DeveloperAt our organization, we are seeking a skilled Web Application Developer to join our dynamic team.As a Web Application Developer, you will be responsible for designing and implementing high-quality web applications using PHP and Laravel framework.Key Responsibilities:Develop and maintain web...

  • Senior Web Design Specialist

    1 week ago

    Alleppey, Kerala, India beBeeWebDesigner Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Web Design ProfessionalWe are seeking a skilled Web Designer to join our team. This full-time remote role involves designing and maintaining websites, working on graphic design projects, developing front-end web interfaces, and ensuring responsive web design.To succeed in this role, you will need:Strong skills in Web Design and Responsive Web...

  • Full Stack Web Application Specialist

    6 days ago

    Alleppey, Kerala, India beBeeWebDeveloper Full time ₹ 6,00,000 - ₹ 10,00,000

    Senior Web Development PositionWe are seeking a highly skilled and experienced Web Developer to join our team in creating custom web applications, themes, and plugins for various clients.The ideal candidate will be proficient in HTML, CSS, JavaScript, and familiar with tools like Elementor, WooCommerce, and ACF.Responsibilities:Write well-designed, testable,...