Junior Web Crawling Engineer

19 hours ago

Alwar, Rajasthan, India Forage AI Full time

We are seeking a Junior Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.

About Forage AI: Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.

Key Responsibilities:

  • Maintain and enhance existing web scraping and data crawling projects.
  • Develop and refine crawlers using Python-based tools and frameworks.
  • Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
  • Clean, validate, and integrate extracted data into downstream storage systems.
  • Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
  • Optimize crawling efficiency and ensure compliance with web crawling best practices.
  • Collaborate with cross-functional teams to improve data acquisition strategies.

Required Skills & Qualifications:

  • Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
  • Strong knowledge of browser automation tools such as Playwright or Selenium.
  • Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
  • Experience in handling anti-scraping challenges and ensuring robust data extraction.
  • Familiarity with distributed scraping techniques and data pipelines.
  • Ability to troubleshoot and optimize web crawlers for performance and reliability.
  • Strong analytical and problem-solving skills with attention to detail.
  • Excellent communication and inter-personal skills.

Other Infrastructure Requirements

Since this is a completely work-from-home position, you will also require the following -

● High-speed internet connectivity for video calls and efficient work.

● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and

no other obstacles to interrupted, efficient work).

● Headphones with clear audio quality.

● Stable power connection and backups in case of internet/power failure.

  • Expertise in Artificial Intelligence and Web Automation Needed

    5 days ago

    Alwar, Rajasthan, India beBeeAutomation Full time ₹ 50,000 - ₹ 70,000

    Job Title: Intelligent Automation SpecialistOverview:We are seeking an experienced automation specialist with strong expertise in artificial intelligence and web crawling to design and build an intelligent system that can interact with websites, perform smart actions, and extract data automatically.Design and develop a scalable AI-powered web automation...

  • Web Data Extraction Specialist

    13 hours ago

    Alwar, Rajasthan, India beBeeExtraction Full time ₹ 9,00,000 - ₹ 12,00,000

    We are seeking a skilled Web Data Extraction Specialist to build and maintain web crawlers, extracting valuable insights from the web.The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.The successful candidate will be responsible for...

  • Global Web Software Engineer

    4 days ago

    Alwar, Rajasthan, India beBeeSoftwareEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    Job OverviewWe are seeking an experienced software engineer to design, develop, and scale secure, high-performance web applications in the travel domain.Design scalable front-end systems with React.js and related frameworks.Develop robust back-end services and APIs using Node.js (Express.js/Nest.js).Integrate external travel APIs and real-time data...

  • Web Engineering Specialist

    4 days ago

    Alwar, Rajasthan, India beBeeDeveloper Full time ₹ 20,00,000 - ₹ 25,00,000

    About Our Engineering Team:We are seeking a skilled engineer who can lead the development of our company website and content-platform stack from start to finish.The ideal candidate will have hands-on experience with Next.js, React, and headless CMS integration. They will be responsible for designing, developing, and shipping new features in Next.js and...

  • Web Application Specialist

    7 days ago

    Alwar, Rajasthan, India beBeeSoftwareDevelopment Full time ₹ 6,00,000 - ₹ 12,00,000

    Role Summary:The Junior Web Developer role is an integral part of our web development and maintenance process. The selected candidate will be responsible for writing, testing, and debugging code.Key Responsibilities:Contribute to front-end and back-end web development tasksCollaborate with cross-functional teamsEnhance web design and functionality through...

  • Expert Web Application Developer Needed

    4 days ago

    Alwar, Rajasthan, India beBeeBackendDeveloper Full time ₹ 10,00,000 - ₹ 20,00,000

    We are seeking a highly skilled and experienced professional to lead our web application projects.Key Responsibilities:Develop scalable web applications using front-end and back-end technologiesLead projects, mentor junior developers, and collaborate with cross-functional teamsEnsure high-quality solutions through strong expertise in software development...

  • Senior Web Development Lead

    4 days ago

    Alwar, Rajasthan, India beBeeSoftware Full time ₹ 15,00,000 - ₹ 25,00,000

    Web Development Lead">We seek a seasoned web development lead to join our dynamic frontend team and drive high-quality web applications.">">Key Responsibilities">Translate designs into responsive UIs, integrating APIs, and optimizing performance, while mentoring junior developers and driving best practices.">">Requirements">5-7 years of hands-on experience...

  • Front-End Web Expert

    5 days ago

    Alwar, Rajasthan, India beBeeDeveloper Full time ₹ 20,00,000 - ₹ 26,00,000

    Web Development Opportunity:Lead front-end development and maintenance of scalable web interfaces using HTML5, CSS3, and Bootstrap.Translate high-fidelity UI/UX designs into functional, mobile-first web components ensuring optimal performance, responsiveness, and accessibility (WCAG compliance).Key ResponsibilitiesDevelop and maintain front-end architecture...

  • Senior Software Engineer

    4 days ago

    Alwar, Rajasthan, India beBeeFullstack Full time ₹ 18,00,000 - ₹ 20,00,000

    Job Title: Full-Stack DeveloperWe are seeking a seasoned Full-Stack Developer to lead our web platform's architecture, scaling, and secure development.This role requires strong expertise in Next.js, Node.js, TypeScript, and MongoDB. The ideal candidate will guide our team, ensure smooth deployments, and handle advanced backend and DevOps challenges.About the...

  • Junior Graphic Designer

    13 hours ago

    Alwar, Rajasthan, India beBeeGraphicDesigner Full time ₹ 40,00,000 - ₹ 70,00,000

    Junior Graphic Designer PositionWe are seeking a creative and motivated Junior Graphic Designer to join our team. The ideal candidate will support the design team in producing engaging visual content for social media, web, and marketing campaigns.The successful candidate will have a strong grasp of design principles, an eye for detail, and a portfolio that...