Senior Web Scraping Engineer

Senior Web Scraping Engineer (Python) — India (Remote)Employment type:Full-time (open to contract-to-hire)Work location:Remote in IndiaTime overlap:Prefer 2–3 hours/day with Pacific Time (PST)About the role We’re building a high-throughput product data ingestion pipeline across hundreds of domains. You’ll own the crawling/extraction layer end‑to‑end: HTTP-first crawling with aPlaywrightfallback, per‑domain learned selectors, and reliable PDF handling (datasheets/specs). You’ll also drive theautomationaround scheduling, retries, and monitoring so runs are hands‑off, and you’ll integrate vendor/publicAPIs (REST/GraphQL)wherever available to complement crawling.This role spanscrawling(discovering & fetching pages via sitemaps/robots) andscraping(extracting structured specs, images, and PDFs into our schema).What you’ll doDesign anHTTP-first crawler(Scrapy or aiohttp) withPlaywright fallbackonly for JS‑heavy pages.Implementsitemap diffingandconditional GETs(ETag/Last-Modified) for incremental runs.Build a lightweight“needs JS?” classifier(HTML length, JSON‑LD presence, data‑product markers) to auto‑route HTTP vs Playwright.Enforceper-domain throttles/backoff(2–4 concurrent/domain; auto‑lower on 429/503).AddURL normalization/canonicalizationand de‑dup (respect; hash PDFs).HandlePDF discovery & download(HEAD first to dedupe; size/concurrency caps; SHA‑256 keys).ApplyPlaywright browser automationresource budgets (block images/fonts/analytics; kill outliers by size/CPU/time).Integrate third‑party APIs(REST/GraphQL) as first‑class sources: handle auth (API keys/OAuth2), pagination, and rate limits;unify API + crawl outputs .Ownautomation & orchestrationfor scheduled runs (Airflow/Temporal/Celery or cron), idempotent retries, and alerting.Createper‑domain selectors(YAML) with verification on hold‑outs; re‑learn only when health drops.Shipobservability : per‑site field coverage, error rates, retries, avg page time, and PDF success.Maintainallow/deny paths ; adhere to robots.txt and Terms of Service.Containerize workers; provide runbooks/CI; collaborate with data team on schemas/normalization.Must‑have qualifications4+ yearsPython , including 2+ years buildingproduction web crawlersat scale.Strong withScrapy or aio Puppeteer) in production.Practicalproxy management , polite anti‑bot tactics, andper‑domain rate limiting .Hands‑on withETag/Last-Modified , retries, backoff, and HTTP caching.Confident withCSS/XPath ,schema.org/JSON‑LD , and HTML parsing.APIs:consumingREST/GraphQL(auth, pagination, backoff) and building small internal services ( FastAPIor similar).Automation/Orchestration:Airflow/Temporal/Celery (or equivalent schedulers/queues) for scheduled runs and monitoring.PDF handling(requests/HEAD, hashing, size limits) and file integrity checks.Queues ( Redis/Kafka ),Docker , Linux basics; comfort with logs/metrics.Clear, pragmatic communication and strong ownership.Nice to haveGoorNode.jsexperience for high‑performance crawlers.Cloud:AWS/GCP ,S3 , ECS/Kubernetes; IaC basics.Workflow engines:Airflow/Temporal/Argo/Celery .Document extraction:Textract/Tika/Camelot/Tabula .Search/analytics:Elasticsearch/OpenSearch ; warehousing ( Snowflake/Postgres ).LLM‑assisted selector generationwith deterministic verification (optional).How we workShip in small, measurable increments.Track coverage and freshness as north‑star metrics.Prefer simple designs that are easy to operate at scale.Compensation Competitive; please include yourexpected CTC (INR LPA)and anyvariable/benefitsexpectations.Application Please apply with your resume and links to relevant repos or code samples. Include concise notes on:a crawler you ran at100+ sites/day(or similar scale),how you handledrate limits/retries , andyour approach toPDF discovery/dedup .