Junior Web Crawling Engineer

4 days ago

India Forage AI Full time
Job Description

We are seeking a Junior Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.

About Forage AI: Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.

Key Responsibilities:

- Maintain and enhance existing web scraping and data crawling projects.
- Develop and refine crawlers using Python-based tools and frameworks.
- Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
- Clean, validate, and integrate extracted data into downstream storage systems.
- Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
- Optimize crawling efficiency and ensure compliance with web crawling best practices.
- Collaborate with cross-functional teams to improve data acquisition strategies.

-

Required Skills & Qualifications:

- Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
- Strong knowledge of browser automation tools such as Playwright or Selenium.
- Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
- Experience in handling anti-scraping challenges and ensuring robust data extraction.
- Familiarity with distributed scraping techniques and data pipelines.
- Ability to troubleshoot and optimize web crawlers for performance and reliability.
- Strong analytical and problem-solving skills with attention to detail.
- Excellent communication and inter-personal skills.

Other Infrastructure Requirements

Since this is a completely work-from-home position, you will also require the following -

High-speed internet connectivity for video calls and efficient work.

Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and

no other obstacles to interrupted, efficient work).

Headphones with clear audio quality.

Stable power connection and backups in case of internet/power failure.

  • Master Web Data Extractor

    3 days ago

    India beBeeData Full time ₹ 7,50,000 - ₹ 15,00,000

    Web Crawling Engineer PositionWe are seeking a skilled Web Crawling Engineer who will be responsible for designing and implementing web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling...

  • Junior Web Developer

    1 week ago

    India Kliniko Ai Full time

    Role DescriptionThe Junior Web Developer will be responsible for participating in both front-end and back-end web development tasks. Daily activities include writing and testing code, troubleshooting and debugging web applications, collaborating with the design and development team, and contributing to web design and functionality enhancement. The Junior Web...

  • Junior Web Developer

    6 days ago

    India Kliniko Ai Full time

    Role Description The Junior Web Developer will be responsible for participating in both front-end and back-end web development tasks. Daily activities include writing and testing code, troubleshooting and debugging web applications, collaborating with the design and development team, and contributing to web design and functionality enhancement. The Junior...

  • Junior Web Developer

    4 weeks ago

    India Trinet Technologies Full time

    Role DescriptionThe Junior Web Developer will be responsible for building and maintaining web applications, working on both front-end and back-end development tasks. Daily responsibilities include coding, designing, and troubleshooting web applications. The role requires close collaboration with other team members to ensure the quality and performance of web...

  • Junior Web Developer

    4 weeks ago

    India The Smart Brief & Co Full time

    Role DescriptionAs a Junior Web Developer, you will be responsible for developing, testing, and maintaining web applications. Day-to-day tasks include working on both front-end and back-end development, collaborating with the design team to implement web designs, and ensuring website functionality and optimization. You will also support programming tasks and...

  • Junior Web Developer

    3 weeks ago

    India Trinet Technologies Full time

    Role DescriptionThe Junior Web Developer will be responsible for building and maintaining web applications, working on both front-end and back-end development tasks. Daily responsibilities include coding, designing, and troubleshooting web applications. The role requires close collaboration with other team members to ensure the quality and performance of web...

  • Junior Web Developer

    4 weeks ago

    India Knowledge Gainer Institute Full time

    Role DescriptionThe Junior Web Developer will be responsible for assisting in the development and maintenance of web applications, collaborating with senior developers to implement front-end designs, and integrating back-end services. The role also includes troubleshooting issues, optimizing website performance, and ensuring a seamless user...

  • Junior Web Developer

    4 weeks ago

    India The Smart Brief & Co Full time

    Role Description As a Junior Web Developer, you will be responsible for developing, testing, and maintaining web applications. Day-to-day tasks include working on both front-end and back-end development, collaborating with the design team to implement web designs, and ensuring website functionality and optimization. You will also support programming tasks...

  • Junior Software Engineer

    3 days ago

    India Varn Infra Developers Pvt Ltd Full time ₹ 5,00,000 - ₹ 10,00,000 per year

    Role DescriptionThe Junior Software Engineer will be responsible for tasks related to software development, back-end web development, and programming. Object-Oriented Programming (OOP) is a key aspect of this role.QualificationsSoftware Development and Programming skills Back-End Web Development skills Proficiency in Object-Oriented Programming (OOP)Degree...

  • Urgent) Junior Web Developer

    4 weeks ago

    India Knowledge Gainer Institute Full time

    Role DescriptionThe Junior Web Developer will be responsible for assisting in the development and maintenance of web applications, collaborating with senior developers to implement front-end designs, and integrating back-end services. The role also includes troubleshooting issues, optimizing website performance, and ensuring a seamless user...