Web Crawling Engineer

4 weeks ago

Mumbai, India Forage AI Full time

We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget upto - Rs. 10 LPAAbout Forage AI : Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale.Key Responsibilities:Maintain and enhance existing web scraping and data crawling projects.Develop and refine crawlers using Python-based tools and frameworks.Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.Clean, validate, and integrate extracted data into downstream storage systems.Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).Optimize crawling efficiency and ensure compliance with web crawling best practices.Collaborate with cross-functional teams to improve data acquisition strategies.Required Skills & Qualifications:Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).Strong knowledge of browser automation tools such as Playwright or Selenium.Solid understanding of HTML, CSS, and selector languages (XPath/CSS).Experience in handling anti-scraping challenges and ensuring robust data extraction.Familiarity with distributed scraping techniques and data pipelines.Ability to troubleshoot and optimize web crawlers for performance and reliability.Strong analytical and problem-solving skills with attention to detail.Excellent communication and inter-personal skills.Other Infrastructure RequirementsSince this is a completely work-from-home position, you will also require the following -● High-speed internet connectivity for video calls and efficient work.● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, andno other obstacles to interrupted, efficient work).● Headphones with clear audio quality.● Stable power connection and backups in case of internet/power failure.

  • Web Crawling Specialist

    1 week ago

    mumbai, India beBeeData Full time

    Web Crawling SpecialistWe are seeking a highly skilled web crawling specialist to join our team. This role requires expertise in building and maintaining complex web crawlers, extracting valuable insights from the web, and ensuring data quality.The ideal candidate will have strong programming skills in Python, experience with web scraping frameworks such as...

  • Web Crawling Specialist

    2 weeks ago

    mumbai, India beBeeCrawling Full time

    Web Crawling SpecialistWe are looking for a skilled expert to develop and maintain web crawlers, extracting valuable insights from the web, and ensuring data quality.About the PositionDesign and build high-performance web crawlers using Python-based tools and frameworks.Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content...

  • Data Engineering Manager – Web Crawling

    3 weeks ago

    Mumbai, India AIMLEAP Full time

    Data Engineering Manager – Web Crawling & Pipeline Architecture Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate  What We Are Looking For Proven...

  • Web Performance

    2 days ago

    Mumbai, Maharashtra, India arrivia Full time

    Are you a developer who sees search engines as the ultimate API? Do you thrive on optimizing the code and architecture of major websites to drive massive organic traffic?We are searching for a Web Performance & SEO Developer to join our global Performance Marketing team. This is a critical hybrid role where you use your developer expertise (.NET, PHP,...

  • Senior Software Engineer

    2 days ago

    Mumbai, Maharashtra, India Nexio Full time

    Senior Software Engineer (Puppeteer/React/NodeJS)  BACHELOR'S DEGREE IN COMPUTER SCIENCE OR SIMILAR ENGINEERING DEGREE REQUIRED, WITH 5+ YEARS OF RELEVANT WORK EXPERIENCE  Qualifications: Hands-on and proficient in ReactJS, , HTML5, CSS3, and JavaScriptExperience in creating responsive web applications and cross-browser compatibilityProven experience with...

  • Search Engine Optimization Manager

    3 weeks ago

    Mumbai, India GoodSpace AI Full time

    SEO - Associate Manager :Responsibilities :The candidate is responsible for enhancing organic visits for our client’s website & to make sure that the goals are achieved.The candidate needs to check information available from various tools and ensure that the suggestions are implemented to enhance the number of organic keywords and to get better ranking for...

  • Search Engine Optimization Manager

    3 weeks ago

    Mumbai, India GoodSpace AI Full time

    SEO - Associate Manager :Responsibilities :The candidate is responsible for enhancing organic visits for our client’s website & to make sure that the goals are achieved.The candidate needs to check information available from various tools and ensure that the suggestions are implemented to enhance the number of organic keywords and to get better ranking for...

  • Search Engine Optimization Manager

    3 weeks ago

    Mumbai, India GoodSpace AI Full time

    SEO - Associate Manager : Responsibilities : The candidate is responsible for enhancing organic visits for our client’s website & to make sure that the goals are achieved. The candidate needs to check information available from various tools and ensure that the suggestions are implemented to enhance the number of organic keywords and to get better ranking...

  • Search Engine Optimization Manager

    3 weeks ago

    Mumbai, India GoodSpace AI Full time

    SEO - Associate Manager : Responsibilities : - The candidate is responsible for enhancing organic visits for our client's website & to make sure that the goals are achieved. - The candidate needs to check information available from various tools and ensure that the suggestions are implemented to enhance the number of organic keywords and to get better...

  • Senior Backend Infrastructure Specialist

    7 days ago

    mumbai, India beBeeAutomation Full time

    Job Opening:We are seeking an innovative and hands-on Full Stack Developer who will help us build internal tools, management consoles, and automation pipelines, enabling self-service capabilities and more efficient operations.You will play a critical role in systemizing crawler lifecycle management—such as launching new crawling instances, monitoring...