Junior Web Crawling Engineer

14 hours ago

Ajmer, Rajasthan, India Forage AI Full time

We are seeking a Junior Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.

About Forage AI: Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.

Key Responsibilities:

  • Maintain and enhance existing web scraping and data crawling projects.
  • Develop and refine crawlers using Python-based tools and frameworks.
  • Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
  • Clean, validate, and integrate extracted data into downstream storage systems.
  • Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
  • Optimize crawling efficiency and ensure compliance with web crawling best practices.
  • Collaborate with cross-functional teams to improve data acquisition strategies.

Required Skills & Qualifications:

  • Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
  • Strong knowledge of browser automation tools such as Playwright or Selenium.
  • Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
  • Experience in handling anti-scraping challenges and ensuring robust data extraction.
  • Familiarity with distributed scraping techniques and data pipelines.
  • Ability to troubleshoot and optimize web crawlers for performance and reliability.
  • Strong analytical and problem-solving skills with attention to detail.
  • Excellent communication and inter-personal skills.

Other Infrastructure Requirements

Since this is a completely work-from-home position, you will also require the following -

● High-speed internet connectivity for video calls and efficient work.

● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and

no other obstacles to interrupted, efficient work).

● Headphones with clear audio quality.

● Stable power connection and backups in case of internet/power failure.

  • Senior Web Data Extraction Specialist

    10 hours ago

    Ajmer, Rajasthan, India beBeeData Full time ₹ 12,00,000 - ₹ 17,00,000

    Web Crawling EngineerWe are seeking a skilled Web Crawling Engineer to build and maintain web crawlers, extracting valuable insights from the web. As a key member of our team, you will be responsible for developing and refining crawlers using Python-based tools and frameworks.Key Responsibilities:Maintain and enhance existing web scraping and data crawling...

  • Junior Web Specialist

    7 days ago

    Ajmer, Rajasthan, India beBeeFrontend Full time ₹ 6,00,000 - ₹ 9,00,000

    Seeking an innovative and detail-oriented professional to join our team as a Junior Web Developer.Key Responsibilities:Contribute to the design, development, and maintenance of web applicationsCollaborate with cross-functional teams to enhance web functionality and designDevelop and test code for web applications, ensuring high-quality resultsIdentify and...

  • AI Automation Developer

    5 days ago

    Ajmer, Rajasthan, India beBeeAutomation Full time ₹ 50,000 - ₹ 70,000

    We are seeking an experienced AI/ML developer to design and develop a robust web automation system capable of interacting with websites intelligently.The ideal candidate will have strong expertise in web automation and a solid understanding of AI/ML techniques, including NLP and reinforcement learning.The successful applicant will be responsible for...

  • Junior Web Development Specialist

    7 hours ago

    Ajmer, Rajasthan, India beBeeWebDeveloper Full time ₹ 50,00,000 - ₹ 80,00,000

    As a key member of the development team, this role is responsible for contributing to both front-end and back-end web development projects.Key responsibilities include writing and testing code, troubleshooting and debugging web applications, collaborating with designers and developers, and enhancing web design and functionality. The successful candidate will...

  • Web Software Engineer

    18 hours ago

    Ajmer, Rajasthan, India beBeeDeveloper Full time ₹ 6,00,000 - ₹ 10,00,000

    Job OpportunityWe are seeking a skilled Web Developer to join our team.This full-time role involves designing, coding, and modifying websites according to company specifications. Tasks include creating and maintaining web applications, collaborating with designers and other team members, debugging and solving technical issues, and ensuring the functionality...

  • High-Performance Data Engineer

    7 days ago

    Ajmer, Rajasthan, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

    A Data Engineer plays a vital role in ensuring data infrastructure can handle increasing volumes of data while maintaining exceptional performance standards. The ideal candidate will design, build, and maintain high-performance data pipelines that process terabytes of data. They'll work on optimizing existing systems, identifying and resolving performance...

  • Dynamic Junior Frontend Developer Opportunities

    7 days ago

    Ajmer, Rajasthan, India beBeeFrontend Full time ₹ 40,00,000 - ₹ 70,00,000

    Opportunities for Junior Frontend DevelopersWe are looking for talented individuals to join our global team as Junior Frontend Developers. As key members of our development team, you will be responsible for building responsive and user-friendly web interfaces using modern frontend frameworks such as React, Vue, or Angular.You will be developing high-quality...

  • Web Solutions Specialist

    5 days ago

    Ajmer, Rajasthan, India beBeeDeveloper Full time ₹ 9,00,000 - ₹ 12,00,000

    Senior Web Developer OpportunityWe are seeking a highly skilled Senior Web Developer to lead our e-commerce development efforts, create scalable solutions, and ensure a seamless user experience across web and mobile.The ideal candidate will have deep expertise in Shopify Liquid, theme customization, and CMS platforms.Design, develop, and optimize Shopify...

  • Advanced Frontend Engineering Specialist

    1 week ago

    Ajmer, Rajasthan, India beBeeDeveloper Full time ₹ 15,00,000 - ₹ 25,00,000

    About UsWe're a leading audio entertainment platform, empowering storytelling through audiobooks, audio series, and podcasts across languages.Key ResponsibilitiesDevelop highly responsive web applications that meet user needs.Collaborate with product managers, designers, and engineers to deliver seamless user experiences.Deliver projects from technical...

  • Senior Test Engineer

    2 days ago

    Ajmer, Rajasthan, India beBeeAutomation Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title: Senior Test EngineerWe are seeking a skilled Senior Test Engineer with expertise in manual and automated testing. The ideal candidate will have deep experience in building scalable automation frameworks, ensuring high-quality product delivery, and mentoring junior team members.Key Responsibilities:Design and develop robust automation frameworks...