Web Crawling Engineer

3 days ago

bangalore, India Forage AI Full time

We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget upto - Rs. 10 LPA About Forage AI : Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale. Key Responsibilities: Maintain and enhance existing web scraping and data crawling projects. Develop and refine crawlers using Python-based tools and frameworks. Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content. Clean, validate, and integrate extracted data into downstream storage systems. Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.). Optimize crawling efficiency and ensure compliance with web crawling best practices. Collaborate with cross-functional teams to improve data acquisition strategies. Required Skills & Qualifications: Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy). Strong knowledge of browser automation tools such as Playwright or Selenium. Solid understanding of HTML, CSS, and selector languages (XPath/CSS). Experience in handling anti-scraping challenges and ensuring robust data extraction. Familiarity with distributed scraping techniques and data pipelines. Ability to troubleshoot and optimize web crawlers for performance and reliability. Strong analytical and problem-solving skills with attention to detail. Excellent communication and inter-personal skills. Other Infrastructure Requirements Since this is a completely work-from-home position, you will also require the following - ● High-speed internet connectivity for video calls and efficient work. ● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and no other obstacles to interrupted, efficient work). ● Headphones with clear audio quality. ● Stable power connection and backups in case of internet/power failure.

  • Web Crawling Specialist

    3 days ago

    bangalore, India beBeeWebCrawler Full time

    Job Opportunity: Web Crawling SpecialistWe are seeking an experienced Web Crawling Specialist to join our team and contribute to the development of innovative web crawling solutions.The ideal candidate will have strong expertise in Python programming, particularly in web scraping frameworks such as Scrapy. Experience with browser automation tools like...

  • Technical Lead – Web Crawling Systems, Data Pipelines

    2 weeks ago

    bangalore, India AIMLEAP Full time

    Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate What We Are Looking For Proven experience leading data engineering teams with strong ownership of web crawling...

  • Expert Web Data Extractor

    1 week ago

    bangalore, India beBeeWebCrawling Full time

    Web Crawling DeveloperWe are seeking a skilled professional to build and maintain web crawlers, extract valuable insights from the web, and ensure data quality.The ideal candidate has strong programming skills in Python and experience with web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Maintain and enhance existing...

  • Senior Web Crawler Specialist

    2 weeks ago

    bangalore, India beBeeEngineer Full time

    We are seeking a skilled engineer to build and maintain web crawlers, extract valuable insights from the web, and ensure data quality.The ideal candidate will have strong programming skills and experience in scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Key Responsibilities:Maintain and enhance existing web scraping...

  • Senior Software Engineer

    1 week ago

    bangalore, India beBeeDataCrawler Full time

    Job OpportunityWe are seeking a seasoned software engineer with expertise in developing scalable web crawling solutions.Key Responsibilities:Design and implement efficient algorithms for data extraction from complex websites, handling dynamic and JavaScript-rendered content.Troubleshoot and resolve issues related to crawl performance, request handling, and...

  • Highly Experienced Python Web Crawler Developer Wanted

    4 days ago

    bangalore, India beBeePython Full time

    Job OpportunityJob SummaryWe are seeking an experienced Senior Python Developer to join our team.This role involves developing and maintaining efficient web crawlers with a focus on speed, performance, and scalability.The ideal candidate will have strong coding skills in Python and a good understanding of how web pages, HTTP, and crawlers work.They should...

  • Senior Data Engineering Lead

    2 weeks ago

    bangalore, India beBeeData Full time

    Job Summary:We are seeking a seasoned professional to lead our data engineering and web crawling teams.The ideal candidate will have extensive experience in architecting, implementing, and optimizing scalable data pipelines.A robust background in building and maintaining crawling systems is essential.The ability to establish pipeline orchestration using...

  • Python Automation and Web Scraping Specialist

    3 weeks ago

    Bangalore Division, India Foresiet Full time

    Company Description Foresiet is an AI-enabled SaaS-based Cybersecurity company that provides a comprehensive solution for Digital Risk Prevention. Leveraging the Cyber Digital Investigator platform, Foresiet proactively detects, monitors, and secures identity, data, and asset threats. Our unique combination of Human Intelligence (HUMINT) and Applied Research...

  • Data Engineering Lead

    2 weeks ago

    bangalore, India beBeeTechnical Full time

    Job TitleA technical architect with hands-on coding skills is required to join our organization. The ideal candidate will have expertise in data/content crawling from the public internet, including bot detection avoidance techniques and VPN use.Key responsibilities include:Developing ETL processes using cloud-native technologiesArchitecting and implementing...

  • Junior Web Developer

    7 days ago

    bangalore, India Web Design Development Jobs US Based Full time

    Company Description Web Design Development Jobs US-Based is a dedicated platform connecting web designers, developers, and graphic design professionals with trustworthy companies seeking top talent. Known for its curated job listings, the platform offers remote, freelance, and full-time opportunities for all experience levels. With new positions posted daily...