Web Crawling Engineer

2 hours ago

india, IN Forage AI Full time

We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.

Salary budget upto - Rs. 10 LPA


About Forage AI: Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.


Key Responsibilities:


  • Maintain and enhance existing web scraping and data crawling projects.
  • Develop and refine crawlers using Python-based tools and frameworks.
  • Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
  • Clean, validate, and integrate extracted data into downstream storage systems.
  • Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
  • Optimize crawling efficiency and ensure compliance with web crawling best practices.
  • Collaborate with cross-functional teams to improve data acquisition strategies.


Required Skills & Qualifications:

  • Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
  • Strong knowledge of browser automation tools such as Playwright or Selenium.
  • Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
  • Experience in handling anti-scraping challenges and ensuring robust data extraction.
  • Familiarity with distributed scraping techniques and data pipelines.
  • Ability to troubleshoot and optimize web crawlers for performance and reliability.
  • Strong analytical and problem-solving skills with attention to detail.
  • Excellent communication and inter-personal skills.


Other Infrastructure Requirements

Since this is a completely work-from-home position, you will also require the following -


● High-speed internet connectivity for video calls and efficient work.

● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and

no other obstacles to interrupted, efficient work).

● Headphones with clear audio quality.

● Stable power connection and backups in case of internet/power failure.

  • Data Engineer

    2 hours ago

    india, IN Alternative Path Full time

    Alternative Path is seeking skilled software developers to collaborate on client projects with an asset management firm. In this role, you will collaborate with individuals across various company departments to shape and innovate new products and features for our platform, enhancing existing ones. You will have a large degree of independence and trust, but...

  • Web Scraping Engineer

    1 hour ago

    Gurugram, Haryana, India, IN noon Full time

    Job title: Web Scraping EngineerLocation: GurgaonAbout the RoleWe are looking for a Web Scraping Engineer with proven experience in building, maintaining, and scaling data extraction systems - especially from e-commerce platforms.The ideal candidate will design and implement robust scrapers to collect, clean, and normalize product data (pricing,...

  • Search Engine Optimization Manager

    1 hour ago

    Gurugram, Haryana, India, IN Loch Corporate Services Full time

    Position Title: SEO ManagerEmployer: Loch Corporate Services Private LimitedLocation: Gurugram, India (Work from office mandatory)Position Type: Full timeReports To: Performance Marketing ManagerRemuneration: Commensurate with experienceAbout the organisationLoch Corporate Services Private Limited is a professional services firm delivering a broad range of...

  • Technical Support Engineer for a Web/Mobile application for US Company

    1 hour ago

    india, IN Textellent, Inc. Full time

    You will work closely with Client Success and support them by providing Tier-2 client support for software technical issues and work with Engineering department to debug and resolve the issues.RESPONSIBILITIES:Provide technical support to internal and external users including customer service teams as well as clients and partnersCommunication and...

  • Mobile/Web QA Engineer

    1 hour ago

    Gurugram, Haryana, India, IN BayOne Solutions Full time

    Responsibilities:· Collaborate with cross-functional teams to understand project requirements· Come up with Test Plan and write Test Cases for Mobile Applications/Web Applications· Execute the Test Cases Manually and provide sign Offs · Define test automation strategies.· Develop and maintain end-to-end test suites using Detox to validate web...

  • AI Engineer Intern

    2 hours ago

    india, IN Qureal AI Full time

    Hiring: AI Engineer Intern (Remote + Paid)We are looking for a dedicated AI Engineer Intern to join our team for a 4-month paid internship. This role offers a real opportunity to transition into a full-time position based on performance. Role Details:Role: AI Engineer InternLocation: RemoteType: Paid Internship (4 Months)Future: PPO / Full-time conversion...

  • SailPoint Engineer

    2 hours ago

    india, IN Imetris Corporation Full time

    Kindly read the job description CAREFULLY before applying. Prefer candidates who can start within 3-4 weeks.Title: SailPoint EngineerPosition: Mid-LevelWork mode: RemoteLocation: Prefer local to Mumbai. 3+ years of hands-on Beanshell development experience with SailPoint IdentityIQ3+ years of hands-on experience with SailPoint IdentityIQ configuration,...

  • Mid QA/Test Engineer

    1 hour ago

    india, IN Imfuna Full time

    Mid QA/Test EngineerAbout usImfuna delivers world-class digital inspection apps and a SaaS-based web report publishing solution for the property and construction markets. Our innovative tools transform inefficient industry processes into streamlined, repeatable, and professional workflows. We leverage cutting-edge mobile and cloud technology to provide...

  • Machine Learning Engineer

    1 hour ago

    india, IN BayOne Solutions Full time

    Responsibilities:Machine Learning Development & Implementation (40%)Design and implement end-to-end ML pipelines for recommendation systems, search ranking, and classification problemsBuild and optimize traditional ML models using techniques such as ensemble methods, SVMs, gradient boosting, and neural networksDevelop time series forecasting models and...

  • Engineering Manager – MEAN/MERN

    2 hours ago

    Bangalore Urban, Karnataka, India, IN Alp Consulting Ltd. Full time

    The ideal candidate will be responsible for managing and inspiring his or her team to achieve their performance metrics. Your role will involve strategizing, project management, part staff management. Your collaborative attitude and interpersonal skills will help you thrive as an Engineering Manager. QualificationsBachelor's degree or equivalent experience...