Web Crawling Engineer

4 days ago


India Forage AI Full time

We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.

Key Responsibilities:

  • Maintain and enhance existing web scraping and data crawling projects.
  • Develop and refine crawlers using Python-based tools and frameworks.
  • Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
  • Clean, validate, and integrate extracted data into downstream storage systems.
  • Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
  • Optimize crawling efficiency and ensure compliance with web crawling best practices.
  • Collaborate with cross-functional teams to improve data acquisition strategies.

Required Skills & Qualifications:

  • Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
  • Strong knowledge of browser automation tools such as Playwright or Selenium.
  • Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
  • Experience in handling anti-scraping challenges and ensuring robust data extraction.
  • Familiarity with distributed scraping techniques and data pipelines.
  • Ability to troubleshoot and optimize web crawlers for performance and reliability.
  • Strong analytical and problem-solving skills with attention to detail.
  • Excellent communication and inter-personal skills.

Other Infrastructure Requirements

Since this is a completely work-from-home position, you will also require the following -

● High-speed internet connectivity for video calls and efficient work.

● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and

no other obstacles to interrupted, efficient work).

● Headphones with clear audio quality.

● Stable power connection and backups in case of internet/power failure.



  • India Forage AI Full time

    Forage AI is seeking a skilled Web Crawling Specialist to join our remote team. As a Web Crawling Specialist, you will play a crucial role in developing and refining crawlers using Python-based tools and frameworks. This involves building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.To excel in this role,...


  • India Adglobal360 Full time

    We are seeking an experienced Senior Software Engineer to join our team as a Web Crawling Expert. In this role, you will be responsible for designing and developing robust web crawling solutions using Python and related libraries.">The ideal candidate will have strong expertise in web scraping and data extraction using Python, proficiency in web scraping...


  • India Hindustan Times Full time

    We are Hindustan Times, a pioneering media house, and we are seeking a talented Python Web Crawling Expert to join our team. Our platform provides valuable insights and analytics to empower decision-makers in the private market.About the JobIn this role, you will design and implement web crawlers in Python for extracting data from diverse online platforms....


  • India YipitData Full time

    Job DescriptionAbout YipitData:YipitData is the leading market research and analytics firm for the disruptive economy and recently raised up to $475M from The Carlyle Group at a valuation over $1B.We analyze billions of alternative data points every day to provide accurate, detailed insights on ridesharing, e-commerce marketplaces, payments, and more. Our...


  • India Petras Solutions Private Limited Full time

    Job Description:Responsible for designing and developing distributed web crawlers that can independently solve various problems encountered in the actual development process.Responsible for researching and developing web page information extraction technology algorithms to improve the efficiency and quality of data capture.Responsible for analyzing and...


  • India YipitData Full time

    About YipitData:We are a fast-growing technology company backed by The Carlyle Group and Norwest Venture Partners. Our offices are located in various cities around the world, but this role can be fully remote based in India with standard work hours from 11am to 8pm IST.We cultivate a people-centric culture focused on mastery, ownership, and transparency. As...


  • India Forage AI Full time

    We are seeking a talented Python Web Scraping Developer to join our team at Forage AI. As a Python Web Scraping Developer, you will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.You will develop and refine crawlers using Python-based tools and frameworks such as Scrapy, and...


  • India Petras Solutions Private Limited Full time

    About the Job : In full growth, particularly internationally, we are looking for new collaborators to join our fabulous team A young but experienced, dynamic and complementary team: a resolutely start-up spirit Real job and career opportunities A friendly atmosphere and a climate of trust that promotes autonomy and challenge" Responsibilities : - Responsible...


  • India Forage AI Full time

    We are seeking a highly skilled Data Extraction Engineer to join our team at Forage AI. In this role, you will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.As a key member of our data acquisition team, you will develop and refine crawlers using Python-based tools and frameworks...


  • India Hindustan Times Full time

    About Us : Hindustan Times, part of Mosaic Digital, is a pioneering company dedicated to creating innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market.Overview : We are seeking talented Software Developers...

  • Data Engineer

    7 days ago


    India HT Digital Streams Full time

    Job Description : Data Engineer (Python). About Us : VCCEdge, part of Mosaic Digital is an upcoming company dedicated to building innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market. Overview: We are seeking an...

  • Data Engineer

    4 days ago


    India Forage AI Full time

    As a Data Engineer at Forage AI, you will be responsible for designing, building, and maintaining scalable data pipelines for data ingestion, processing, and storage. You will work closely with cross-functional teams to integrate crawled data from external sources and ensure seamless data flow. Your role will involve implementing robust monitoring and...

  • Software Engineer

    7 days ago


    India Hindustan Times Full time

    We are Hindustan Times, a renowned media house, and we are seeking a talented Software Engineer to join our team. Our platform provides valuable insights and analytics to empower decision-makers in the private market.About UsWe are an upcoming company dedicated to building innovative products for private market stakeholders. Our mission is to provide...

  • Software Developer

    7 days ago


    India Hindustan Times Full time

    About Us : VCCEdge, part of Mosaic Digital is an upcoming company dedicated to building innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market. Overview : We are seeking talented Software Developers proficient in...

  • Software Engineer

    4 weeks ago


    India Forage AI Full time

    Role Summary: Join our dynamic and innovative team dedicated to delivering cutting-edge solutions in web data extraction, AI integration, and software engineering. As a Software Engineer, you will collaborate with a talented group of engineers and data scientists to develop robust, scalable solutions while acting as a crucial link between multiple teams to...

  • Data Scientist

    4 days ago


    India Adglobal360 Full time

    We are seeking a highly skilled Data Scientist to join our team as a Web Scraping Specialist. In this role, you will be responsible for designing and developing robust web scraping solutions using Python and related libraries.">The ideal candidate will have strong expertise in web scraping and data extraction using Python, proficiency in web scraping...


  • India MAGELLANIC CLOUD LIMITED Full time

    Responsibilities : - Design, deploy, and manage highly scalable and reliable search platforms using Lucidworks Fusion and Apache Solr. - Configure and maintain search clusters on cloud environments (GKE, AWS EC2). - Ensure high availability, fault tolerance, and disaster recovery for search infrastructure. - Analyze search performance metrics and identify...


  • India YipitData Full time

    You will be responsible for refactoring and maintaining existing web scrapers, implementing advanced scraping techniques, and collaborating with cross-functional teams. You will also monitor and troubleshoot scraper performance, develop robust monitoring solutions, and propose new tooling and methodologies to enhance our scraping capabilities and...


  • India AviinTech Business Solutions Full time

    Key Responsibilities- Perform technical SEO audits and provide recommendations for site improvements- Optimize website structure and internal linking for improved crawlability- Manage XML sitemaps and robots.txt to ensure proper indexing and crawling- Implement structured data markup to enhance search results appearance- Monitor and analyze site performance...


  • India Forage AI Full time

    Job OverviewWe are seeking a highly motivated and detail-oriented Data Engineer Intern to join our team at Forage AI. As a Data Engineer Intern, you will work closely with our internal teams to develop and integrate AI-based data solutions.ResponsibilitiesCollaborate with engineers to build and optimize LLM-based applications for data extraction, XPath...