Web Crawling Engineer

5 days ago

Delhi, India Forage AI Full time

We are seeking aWeb Crawling Engineerwho will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget upto - Rs. 9 LPAAbout Forage AI : Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.Key Responsibilities:Maintain and enhance existing web scraping and data crawling projects.Develop and refine crawlers using Python-based tools and frameworks.Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.Clean, validate, and integrate extracted data into downstream storage systems.Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).Optimize crawling efficiency and ensure compliance with web crawling best practices.Collaborate with cross-functional teams to improve data acquisition strategies.Required Skills & Qualifications:Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).Strong knowledge of browser automation tools such as Playwright or Selenium.Solid understanding of HTML, CSS, and selector languages (XPath/CSS).Experience in handling anti-scraping challenges and ensuring robust data extraction.Familiarity with distributed scraping techniques and data pipelines.Ability to troubleshoot and optimize web crawlers for performance and reliability.Strong analytical and problem-solving skills with attention to detail.Excellent communication and inter-personal skills.Other Infrastructure RequirementsSince this is a completely work-from-home position, you will also require the following -● High-speed internet connectivity for video calls and efficient work.● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, andno other obstacles to interrupted, efficient work).● Headphones with clear audio quality.● Stable power connection and backups in case of internet/power failure.

  • Web crawling engineer

    4 days ago

    Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Web Crawling Engineer

    2 weeks ago

    Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Web Crawling Engineer

    5 days ago

    Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget...

  • Web Crawling Engineer

    7 days ago

    New Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Web Crawling Engineer

    4 days ago

    Delhi, India Forage AI Full time

    We are seeking aWeb Crawling Engineerwho will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Web Crawling Engineer

    2 weeks ago

    New Delhi, India Forage AI Full time

    We are seeking aWeb Crawling Engineerwho will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget...

  • Python Developers Web Scraping, Crawling

    1 week ago

    Bengaluru, Delhi, Mumbai, NCR, India Hatchtra Innotech Full time ₹ 2,50,000 - ₹ 12,00,000 per year

    Were Hiring: Python Developers Web Scraping, Crawling & Automation Experts(40 Openings | Yrs Experience | Multiple Levels & Roles)Location: Mumbai / Pune / Chandigarh / Bengaluru / Gurugram / Noida / New Delhi (WFO, Hybrid, or Remote Based on RoleIndustry: Fortune 500 Client Projects (Staffing via Hatchtra Innotech Pvt. Ltd.) Employment Type: Full-Time |...

  • Ai And Data Crawling Programmer

    2 weeks ago

    Delhi, India Whatjobs IN C2 Full time

    Job Overview We are seeking a curious and tech-savvy AI and Data Crawling Programmer to support ourdata-driven projects. The ideal candidate will have foundational knowledge in artificial intelligence andmachine learning, with a keen interest in web data extraction. Experience in website development is aplus, as it will aid in understanding site structures...

  • Lead Data Crawling Engineer

    1 day ago

    Delhi, India NuvoRetail Full time

    Position: Lead Data Crawling Engineer Location: Sector 23 Dwarka, Delhi This is a Delhi-based position and work from office only!! Job Summary: We are looking for an experienced Lead Data Crawling Engineer with 4–6 years of hands-on expertise in designing, optimizing, and scaling data extraction systems. The ideal candidate should be highly skilled in...

  • Lead Data Crawling Engineer

    2 days ago

    Delhi, India NuvoRetail Full time

    Position: Lead Data Crawling Engineer Location: Sector 23 Dwarka, Delhi This is a Delhi-based position and work from office only!! Job Summary: We are looking for an experienced Lead Data Crawling Engineer with 4–6 years of hands-on expertise in designing, optimizing, and scaling data extraction systems. The ideal candidate should be highly skilled in...