Web Crawling and Scraping Expert

7 days ago

Mumbai, Maharashtra, India beBeeDataIngestion Full time ₹ 20,00,000 - ₹ 25,00,000
Product Data Ingestion Specialist

We're building a high-throughput pipeline to ingest product data from hundreds of domains. This role encompasses crawling (discovering and fetching pages via sitemaps/robots) and scraping (extracting structured specs, images, and PDFs into our schema).

Key Responsibilities
  • Design a scalable HTTP crawler with Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs (ETag/Last-Modified) for incremental runs.
  • Develop a lightweight classifier to auto-route HTTP vs Playwright based on page requirements.
  • Enforce per-domain throttling/backoff and URL normalization/canonicalization.
  • Add URL de-duplication and handle PDF discovery and download.
  • Apply Playwright browser automation resource budgets and integrate third-party APIs.
  • Own automation and orchestration for scheduled runs and idempotent retries.
  • Ship observability and maintain allow/deny paths.
Must-Have Qualifications
  • 4+ years Python experience, including 2+ years building production web crawlers at scale.
  • Strong skills with Scrapy or aiohttp/asyncio and Playwright in production.
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery for scheduled runs and monitoring.
Nice to Have
  • Go or Node.js experience for high-performance crawlers.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics.
  • Workflow engines: Airflow/Temporal/Argo/Celery.

This is an exciting opportunity to work on a high-throughput product data ingestion pipeline. If you have expertise in web crawling, scraping, and API integration, we'd love to hear from you.

  • Web Scraping Engineer

    1 week ago

    Mumbai, Maharashtra, India Volody Product Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    We are seeking an experienced Web Scraping Engineer with deep expertise in Scrapy to develop, maintain, and optimize web crawlers. The ideal candidate will have a strong background in extracting, processing, and managing large-scale web data efficiently. Responsibilities : Write and maintain web scraping scripts using Python Optimize custom web scraping...

  • Senior Data Extraction Specialist

    2 weeks ago

    Mumbai, Maharashtra, India beBeeWebCrawlerEngineer Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Web Crawler EngineerWe are seeking a skilled Web Crawler Engineer to join our team. As a key member, you will be responsible for designing and developing web crawlers that efficiently extract valuable insights from the web.About this role:Maintain and enhance existing web scraping projects.Develop and refine crawlers using Python-based tools and...

  • Développeur web

    6 days ago

    Mumbai, Maharashtra, India 360 Space Full time

    LONG-TERM FREELANCE - Expert Web Scraping/Data ExtractionWe are looking for a high-level scraper / data automation expert ready to take on a long-term project with serious ambition. We're talking about a smart, evolving system that requires advanced technical skill, reliability, and creativity to bypass complex data access challenges. About the mission...

  • Data Extraction Expert

    1 week ago

    Mumbai, Maharashtra, India beBeeSpecialist Full time ₹ 15,00,000 - ₹ 25,00,000

    Web Scraping SpecialistAs a skilled Web Scraping Specialist, you will be responsible for designing and implementing efficient and scalable data scraping systems using tools like Python, Selenium, BeautifulSoup, Scrapy, and Pandas. Your primary goal will be to extract structured and unstructured data from various websites and APIs.Key Responsibilities:Design,...

  • Data Extraction Expert

    1 week ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 8,00,000 - ₹ 12,00,000

    Job Title: Data Mining SpecialistThis is a full-time position that requires expertise in automating data extraction processes from web platforms to drive business growth.Key Responsibilities:Design, develop, and maintain robust web scraping solutions to extract structured and unstructured data from various websites and APIs.Utilize tools like Python,...

  • Junior Data Extraction Specialist

    2 weeks ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 9,00,000 - ₹ 12,00,000

    Web Scraping Developer RoleThis is a challenging and exciting role for a Junior Web Scraping Developer who is passionate about web scraping and data extraction.We are looking for someone with a strong understanding of HTML, DOM, and browser behavior to join our team. The ideal candidate will have hands-on experience with requests, Selenium, BeautifulSoup,...

  • Junior Python Developer

    2 weeks ago

    Mumbai, Maharashtra, India WeAssemble Full time

    Junior Python Developer - Web ScrapingMumbai, MaharashtraWork Type : Full TimeWere looking for a Junior Python Developer who is passionate about web scraping and data extraction.If you love automating the web, navigating anti-bot mechanisms, and writing clean, efficient code, this role is for youKey Responsibilities :- Design and build robust web scraping...

  • Python Developer

    2 weeks ago

    Mumbai, Maharashtra, India Softcell Technologies Full time US$ 90,000 - US$ 1,20,000 per year

    Role & responsibilitiesTechnical Skills:Proficiency in Python and libraries like BeautifulSoup, Scrapy, and Selenium.• Experience with regular expressions (Regex) for data parsing.• Strong knowledge of HTTP protocols, cookies, headers, and user-agent rotation.• Familiarity with databases (SQL and NoSQL) for storing scraped data. • Hands-on experience...

  • Python Developer

    2 weeks ago

    Mumbai, Maharashtra, India KANALYTICS Full time ₹ 4,00,000 - ₹ 6,00,000 per year

    Python Developer – Data Scraping, MongoDB, Solr / ElasticSearchWe are seeking a skilled Python Developer with strong experience in web/data scraping and working knowledge of MongoDB, Solr, and/or ElasticSearch. You will be responsible for developing, maintaining, and optimizing scalable scraping scripts to collect structured and unstructured data,...

  • Python Developer

    2 weeks ago

    Mumbai, Maharashtra, India kailasa analytics & services Full time ₹ 4,21,115 - ₹ 8,47,231 per year

    We are seeking a skilled Python Developer with strong experience in web/data scraping and working knowledge of MongoDB, Solr, and/or ElasticSearch. You will be responsible for developing, maintaining, and optimizing scalable scraping scripts to collect structured and unstructured data, efficiently manage it in MongoDB, and index it for search and retrieval...