Senior Web Crawling Specialist

2 weeks ago

Alwar, Rajasthan, India beBeeWebScraping Full time ₹ 80,00,000 - ₹ 1,00,00,000

Job Summary:

We are seeking a skilled Senior Web Scraping Engineer to join our team. As a key member of our engineering organization, you will design and implement scalable web crawling systems that can handle complex web pages.

Key Responsibilities:

  • Crawling and Extraction Layer
    • We are looking for someone with expertise in designing HTTP-first crawlers with Playwright fallbacks for JavaScript-heavy pages.
    • The ideal candidate should have experience in implementing sitemap diffing and conditional GETs for incremental runs.
    • We also need someone who can build lightweight classifiers to auto-route HTTP vs Playwright traffic.
    • A strong understanding of per-domain throttles/backoff and URL normalization/canonicalization is required.
    • Addition of URL de-duplication and PDF handling with SHA-256 keys would be an added bonus.
  • Automation and Orchestration
    • We need someone who can integrate third-party APIs as first-class sources with authentication, pagination, and rate limits.
    • Experience in owning automation & orchestration for scheduled runs with Airflow/Temporal/Celery or cron is necessary.
    • The ideal candidate should create per-domain selectors with verification on hold-outs and re-learning only when health drops.
    • Maintaining allow/deny paths and adhering to robots.txt and Terms of Service is crucial.
  • Oversight and Maintenance
    • Ship observability metrics: per-site field coverage, error rates, retries, average page time, and PDF success.
    • Containerizing workers, providing runbooks/CI, and collaborating with the data team on schemas/normalization is essential.
    Requirements:
    • 4+ years of Python experience, including 2+ years of building production web crawlers at scale.
    • Strong knowledge of Scrapy, aiohttp/asyncio, and Playwright in production environments.
    • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting skills are required.
    • Hands-on experience with ETag/Last-Modified, retries, backoff, and HTTP caching is essential.
    • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
    • APIs: consuming REST/GraphQL (auth, pagination, backoff) and building small internal services (FastAPI or similar).
    • Automation/Orchestration: Airflow/Temporal/Celery (or equivalent schedulers/queues) for scheduled runs and monitoring.
    • PDF handling (requests/HEAD, hashing, size limits) and file integrity checks.
    • Queues (Redis/Kafka), Docker, Linux basics; comfort with logs/metrics.

Benefits:

We offer a competitive salary package, comprehensive benefits, and opportunities for professional growth and development.

Others:

We are an equal opportunity employer and welcome applications from diverse candidates.

  • Junior Web Crawling Engineer

    2 weeks ago

    Alwar, Rajasthan, India Forage AI Full time

    We are seeking a Junior Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. About...

  • Web Data Extraction Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeExtraction Full time ₹ 9,00,000 - ₹ 12,00,000

    We are seeking a skilled Web Data Extraction Specialist to build and maintain web crawlers, extracting valuable insights from the web.The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.The successful candidate will be responsible for...

  • Data Ingestion Pipeline Manager

    7 days ago

    Alwar, Rajasthan, India beBeeExpert Full time US$ 70,000 - US$ 1,20,000

    Product Data Ingestion SpecialistWe are building a high-throughput product data ingestion pipeline across hundreds of domains.You will own the crawling/extraction layer end-to-end, leveraging HTTP-first crawling with a Playwright fallback and per-domain learned selectors. Reliable PDF handling for datasheets/specs is also a key aspect of this role.This role...

  • Senior Web Application Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeFrontEnd Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    As a seasoned and accomplished Senior Frontend Developer, you will spearhead the development of high-performance web applications.

  • Web Affiliate Marketing Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeAffiliate Full time ₹ 6,00,000 - ₹ 10,00,000

    Web Affiliate Marketing SpecialistWe are seeking a talented Web Affiliate Marketing Specialist to join our team. The ideal candidate will be responsible for managing, growing, and optimizing our affiliate and publisher network to drive quality leads, conversions, and revenue for our web campaigns (CPL/CPA/CPS models).Key Responsibilities:Identify, recruit,...

  • Senior Web Solutions Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeewebDeveloper Full time ₹ 8,00,000 - ₹ 15,00,000

    Web Developer OpportunityWe are seeking a skilled Web Developer to join our team. The ideal candidate will be responsible for designing, coding, and modifying scalable software solutions tailored for businesses.About the RoleThis full-time remote opportunity involves creating user-friendly web pages, maintaining and improving websites, optimizing...

  • Senior Web Development Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeSoftware Full time US$ 1,50,000 - US$ 2,50,000

    Job PostingWe are seeking an experienced software professional to design, develop, and maintain scalable web applications for our edtech platform.About the RoleThe ideal candidate will have strong expertise in PHP frameworks, databases, and web technologies, with a passion for building robust solutions that enhance user experience.Main...

  • Data Architecture Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time US$ 1,50,000 - US$ 1,75,000

    Senior Data Engineer Position OverviewWe are seeking an exceptional Senior Data Engineer to join our team. As a key member, you will design, build, and maintain high-performance data pipelines that process large volumes of data.About the RoleThis critical role involves optimizing existing systems, identifying performance bottlenecks, and implementing...

  • Senior Web Development Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeWebdeveloper Full time ₹ 8,00,000 - ₹ 12,00,000

    Full Stack Developer OpportunityCreatikartta offers innovative marketing solutions that combine the synergy between marketing and sales. Our goal is to provide authentic service provision through our unique approach, ensuring exceptional results.Job OverviewThis is a full-time on-site opportunity for an experienced Full Stack Developer. Primary...

  • Senior Web Interface Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeUI Full time ₹ 15,00,000 - ₹ 25,00,000

    Job DescriptionAs a seasoned UI Developer, you will play a pivotal role in crafting visually stunning and user-friendly web interfaces. Collaborating closely with UX designers, backend developers, and product teams, you will translate design wireframes into interactive applications.The ideal candidate should have strong expertise in modern front-end...