Web Scraping Specialist

1 day ago

Coimbatore, Tamil Nadu, India beBeeWebScraping Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

This is a challenging opportunity for a Senior Web Scraping Engineer to design and implement an HTTP-first crawler with Playwright fallback, handling complex web pages and integrating third-party APIs.

Key Responsibilities
  • Design and implement a robust HTTP-first crawler with Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing, conditional GETs, and per-domain throttles/backoff to ensure efficient crawling.
  • Build a lightweight 'needs JS?' classifier to auto-route HTTP vs Playwright requests.
  • Handle PDF discovery and download, URL normalization/canonicalization, and de-duplication to ensure data accuracy.
  • Integrate third-party APIs as first-class sources, own automation and orchestration, create per-domain selectors, and ship observability metrics.
RequirementsMust-have qualifications
  • 4+ years of experience in Python programming language.
  • Strong expertise in Scrapy or aiohttp/asyncio and Playwright libraries.
  • Practical proxy management and polite anti-bot tactics to avoid website blocking.
  • Hands-on experience with ETag/Last-Modified, retries, backoff, and HTTP caching mechanisms.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing techniques.
  • APIs: consuming REST/GraphQL and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery for scheduled runs and monitoring.
  • PDF handling and file integrity checks to ensure data accuracy.
  • Queues, Docker, Linux basics; comfort with logs/metrics for troubleshooting.
  • Clear, pragmatic communication and strong ownership to drive project success.
Nice to have
  • Go or Node.js experience for high-performance crawlers.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics for infrastructure management.
  • Workflow engines: Airflow/Temporal/Argo/Celery for task automation.
  • Document extraction: Textract/Tika/Camelot/Tabula for unstructured data processing.
  • Search/analytics: Elasticsearch/OpenSearch; warehousing (Snowflake/Postgres) for data storage.
  • LLM-assisted selector generation with deterministic verification (optional).
What We Offer
  • A dynamic work environment with opportunities for growth and learning.
  • Collaborative team culture with open communication channels.
  • Competitive compensation and benefits package.
  • Flexible work arrangements to balance work-life harmony.

  • Web Scraping Specialist

    4 days ago

    Coimbatore, Tamil Nadu, India SmartStream Full time ₹ 4,00,000 - ₹ 8,00,000 per year

    Job Title: Web Scraping SpecialistExperience: 3 - 6 YearsLocation: Remote (Work from Home)About the jobWe are seeking a highly skilled Web Scraping Specialist to join our team. The successful candidate will be responsible for designing, implementing, and maintaining web scraping processes to gather data from various online sources efficiently and accurately....

  • Senior Web Scraping Specialist

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Lead Data EngineerWe are seeking a highly skilled and experienced professional to join our team as a Lead Data Scraping Engineer. The ideal candidate will have a minimum of 4 years of hands-on experience in IT scraping, with at least 2 years leading a team of 5+ developers.This role requires deep technical knowledge in advanced scraping...

  • Expert Web Scraping and Data Automation

    5 days ago

    Coimbatore, Tamil Nadu, India beBeeAutomation Full time ₹ 9,00,000 - ₹ 12,00,000

    Web Scraping and Data Automation ExpertWe are seeking a seasoned professional to lead the development of a sophisticated data extraction system. This high-level project involves creating a smart, evolving system that requires advanced technical expertise, reliability, and creativity to bypass complex data access challenges.About the ProjectA strategic data...

  • Senior Web Development Specialist

    5 days ago

    Coimbatore, Tamil Nadu, India beBeeCrawling Full time ₹ 9,00,000 - ₹ 12,00,000

    We are currently seeking a skilled professional to fill the role of Web Crawling Developer.Job DescriptionThe successful candidate will be responsible for designing and implementing efficient web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will possess strong Python programming skills and experience...

  • Data Insight Specialist

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 40,00,000 - ₹ 80,00,000

    **Job Title:** Data Insight SpecialistThis is a data analysis position that involves working with large datasets to extract insights and knowledge. The role requires expertise in Python, Selenium, Pandas, SQL, and APIs, with the ability to design and implement efficient and scalable data scraping systems.The ideal candidate will have experience with web...

  • Senior Web Data Extraction Specialist

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeAutomation Full time ₹ 8,00,000 - ₹ 12,00,000

    Job TitleWe are seeking an experienced Python Developer with proven expertise in Scrapy and strong skills in web scraping and automation. The ideal candidate will design, develop, and optimize large-scale data extraction solutions that power business decisions.Design, develop, and maintain scalable web scraping frameworks using Scrapy.Work with additional...

  • Senior Data Specialist

    3 days ago

    Coimbatore, Tamil Nadu, India beBeeDataSpecialist Full time ₹ 90,00,000 - ₹ 1,20,00,000

    As a pioneering data professional, you will have the unique opportunity to contribute directly to foundational development and establish best practices.This role is ideal for someone who thrives on early-stage challenges, loves building innovative, scalable solutions from day zero, and has a strong passion for web scraping and data collection.Web Scraping &...

  • Data Development Specialist

    6 days ago

    Coimbatore, Tamil Nadu, India beBeeSoftware Full time ₹ 9,00,000 - ₹ 12,00,000

    At an asset management firm, we are seeking skilled software developers to collaborate on client projects.This is an opportunity to contribute and help grow a group of like-minded professionals. You will be given ownership and expected to make your voice heard.Collaborate with analysts to understand and anticipate requirements.Design, implement, and maintain...

  • Senior Web Automation Specialist

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeWebAutomation Full time ₹ 6,00,000 - ₹ 7,00,000

    Web Automation EngineerWe are seeking a skilled web automation engineer to design and build an intelligent system that can interact with websites, perform smart actions, and extract data automatically.About the Job:The ideal candidate will have expertise in web automation, AI/ML, and software development. They will be responsible for designing and...

  • Senior Data Extraction Specialist

    5 days ago

    Coimbatore, Tamil Nadu, India beBeeDataExtractor Full time ₹ 8,00,000 - ₹ 15,00,000

    We are seeking a highly skilled professional to design and optimize data extraction solutions.Key Responsibilities:Develop scalable Python scripts for web scraping from structured and unstructured sources.Implement text/data extraction workflows using OCR tools and libraries.Collaborate with teams to integrate extracted data into applications or...