Web Crawling Engineer

5 days ago

Panipat, India Forage AI Full time

We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget upto - Rs. 10 LPA About Forage AI : Forage AI is a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. Trusted by global clients in finance, real estate, and healthcare, Forage AI enables businesses to automate workflows, reduce manual rework, and access high-quality data at scale. Key Responsibilities: Maintain and enhance existing web scraping and data crawling projects. Develop and refine crawlers using Python-based tools and frameworks. Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content. Clean, validate, and integrate extracted data into downstream storage systems. Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.). Optimize crawling efficiency and ensure compliance with web crawling best practices. Collaborate with cross-functional teams to improve data acquisition strategies. Required Skills & Qualifications: Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy). Strong knowledge of browser automation tools such as Playwright or Selenium. Solid understanding of HTML, CSS, and selector languages (XPath/CSS). Experience in handling anti-scraping challenges and ensuring robust data extraction. Familiarity with distributed scraping techniques and data pipelines. Ability to troubleshoot and optimize web crawlers for performance and reliability. Strong analytical and problem-solving skills with attention to detail. Excellent communication and inter-personal skills. Other Infrastructure Requirements Since this is a completely work-from-home position, you will also require the following - ● High-speed internet connectivity for video calls and efficient work. ● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and no other obstacles to interrupted, efficient work). ● Headphones with clear audio quality. ● Stable power connection and backups in case of internet/power failure.

  • SEO Executive

    4 days ago

    Panipat, Haryana, India Synergylabs Technology Full time ₹ 2,40,000 - ₹ 3,00,000 per year

    SEO Executive (Technical SEO)Company: The Shivalika RugsLocation: PanipatExperience: 1–2 YearsJob Type: Full-TimeAbout the Role We are looking for an SEO Executive who understands both regular and technical SEO. You will help improve our website's visibility on Google and other search engines. The ideal candidate should understand how websites work, how to...

  • Full-Stack / Python Developer (Web Scraping & Automation Specialist)

    1 week ago

    Panipat, India Youngun Full time

    Location: 100% Remote Schedule: Flexible, results-driven environment Engagement: Long-term opportunity About MeldIt MeldIt develops large-scale social media automation and data intelligence solutions. Our platform manages campaigns across multiple brands and collects high-volume data from various social platforms. We focus on building robust,...

  • Frontend Engineer

    1 week ago

    Panipat, India AI Library Full time

    Role: Frontend Engineer / Senior Frontend Engineer (Next.Js) Location: India (Gurgaon - On-site) Employment: Full-time About AI Library AI Library builds enterprise-grade agentic AI systems that help organisations automate workflows, make faster decisions, and improve operational efficiency. We work with Fortune 500 companies, large Indian enterprises, and...

  • Customer Engineer

    7 days ago

    Panipat, India Lepton Software Full time

    About Lepton Software: Lepton Software is a leading provider of Location Intelligence, Smart Mapping, and GIS-based software solutions, empowering enterprises across Telecom, BFSI, Government, Retail, and Automotive sectors. With over 25+ years of expertise, Lepton delivers innovative geospatial products, network planning solutions, and enterprise-grade...

  • Senior Software Engineer

    2 weeks ago

    Panipat, India BigStep Technologies Full time

    Job Description: We’re looking for a Full Stack Engineer with 6+ years of experience , strong backend proficiency, and a proven track record in building modern, scalable web applications. The ideal candidate should be highly skilled in Node.js, React.js, relational and NoSQL databases, microservices architecture, and cloud-native development . Hands-on...

  • QA Automation Architect

    1 week ago

    Panipat, India HireMee Full time

    Airlines, Aviation, Defence Company Name : Others Skills: Job Location : Panipat - No. of Openings: - **Not updated** Gender Preference :No Preference **Full Time** APPLY Qualification - Mandatory Qualification: - UG-Non Engineering- Qualifications: - Not Updated- Additional Prefered Skills: - Not UpdatedComplete Job Description Designation - QA...

  • Frontend Developer

    2 weeks ago

    panipat, India Group Bayport Full time

    About Group BayportGroup Bayport is a rapidly growing global e-Commerce, B2B2C, B2B and B2C organization and an industry leader in print technology, operates in the United States, Canada, Australia, New Zealand, UK, and India through its family of brands – Banner Buzz, Covers & All, Vivyx Printing, Circle One, Giant Media, and Neon Earth. From full suites...

  • Mobile Automation Engineer

    3 days ago

    Panipat, India Shipsy Full time

    Role Overview Looking for an SDET to rapidly convert existing manual Android/iOS flows into reliable, scalable automation. You’ll own critical-path coverage, wire suites into CI/CD, and raise release confidence—primarily coding in TypeScript/JavaScript . Responsibilities Design & build automation: Implement and maintain mobile E2E tests for Android/iOS...

  • Director of Engineering

    1 week ago

    Panipat, India Lytegen Full time

    Job Description We are hiring a Director of Engineering to design, build, and scale the full technology foundation for our rapidly growing U.S.-based solar operations and sales organization. This is a critical senior technical leadership role for someone who can understand business needs at a deep level and transform them into clean, automated, scalable...

  • Shopify Senior Developer

    1 week ago

    Panipat, India GMG Full time

    Who we are: GMG is a global well-being company retailing, distributing and manufacturing a portfolio of leading international and home-grown brands across sport, everyday goods, health and beauty, properties and logistics sectors. Under the ownership and management of the Baker family for over 45 years, GMG is a valued partner of choice for the world's most...