Web Crawling Specialist

3 hours ago

Allahabad, Uttar Pradesh, India beBeeWebCrawling Full time ₹ 6,00,000 - ₹ 12,00,000
Job Opportunity

We are seeking a skilled Web Crawling Specialist to join our team. This individual will be responsible for designing, developing, and maintaining web crawlers, as well as extracting valuable insights from the web. The ideal candidate will have strong programming skills in Python and experience with web scraping frameworks, browser automation tools, and anti-scraping mechanisms.

About Us

We are a pioneering AI-powered data extraction and automation company that transforms complex, unstructured web and document data into clean, structured intelligence. Our platform combines web crawling, NLP, LLMs, and agentic AI to deliver highly accurate firmographic and enterprise insights across numerous domains. We work with global clients in finance, real estate, and healthcare, enabling businesses to automate workflows, reduce manual rework, and access high-quality data at scale.

Key Responsibilities:
  • Maintain and enhance existing web scraping and data crawling projects.
  • Develop and refine crawlers using Python-based tools and frameworks.
  • Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
  • Clean, validate, and integrate extracted data into downstream storage systems.
  • Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
  • Optimize crawling efficiency and ensure compliance with web crawling best practices.
  • Collaborate with cross-functional teams to improve data acquisition strategies.
Requirements

To succeed in this role, you will need:

  • Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
  • Strong knowledge of browser automation tools such as Playwright or Selenium.
  • Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
  • Experience in handling anti-scraping challenges and ensuring robust data extraction.
  • Familiarity with distributed scraping techniques and data pipelines.
  • Ability to troubleshoot and optimize web crawlers for performance and reliability.
  • Strong analytical and problem-solving skills with attention to detail.
  • Excellent communication and inter-personal skills.
Infrastructure Requirements

This is a remote position, so you will require:

  • High-speed internet connectivity for video calls and efficient work.
  • A capable business-grade computer (e.g., modern processor, 8 GB+ of RAM).
  • Headphones with clear audio quality.
  • A stable power connection and backups in case of internet/power failure.

  • Senior Web Automation Specialist

    4 days ago

    Allahabad, Uttar Pradesh, India beBeeAutomation Full time ₹ 50,000 - ₹ 70,000

    Job Title: Senior AI DeveloperWe are seeking a seasoned AI/ML expert with expertise in web automation to design and build an intelligent system that can interact with websites, perform smart actions (clicking, downloading, form filling, navigation), and extract data automatically.The role involves combining AI techniques with web crawling/automation tools to...

  • Senior AI Developer

    4 days ago

    Allahabad, Uttar Pradesh, India Stamens Software Pvt. Ltd. Full time

    Job Title: Senior AI Developer-Intelligent Web Automation Location: Fully Remote Employment Type: Full Time/Contract/Freelancer Initial Contract Duration: 6 months — may be extended Experience: 3+ years in AI + automation Budget: 50 LPA – 70 LPA  Position Overview: We are looking for an senior experienced AI/ML Developer with strong expertise...

  • Senior Intelligent Automation Expert

    4 days ago

    Allahabad, Uttar Pradesh, India beBeeWebAutomationDeveloper Full time ₹ 50,000 - ₹ 70,000

    Web Automation Developer Job DescriptionWe are seeking an experienced Web Automation Developer to join our team. The ideal candidate will have strong expertise in web automation and AI/ML techniques.The role involves designing and building intelligent systems that can interact with websites, perform smart actions, and extract data automatically.The...

  • Senior Web Automation Specialist

    3 hours ago

    Allahabad, Uttar Pradesh, India beBeeAutomation Full time ₹ 80,00,000 - ₹ 1,20,00,000

    We are seeking a skilled Senior Web Automation Specialist to develop and maintain automated test scripts using Playwright (JavaScript/TypeScript).Key Responsibilities:Perform functional, regression, and smoke testing of web applications.Collaborate with developers and product teams to understand features and requirements.Identify, document, and track bugs...

  • Cloudflare Web Application Firewall Specialist

    3 days ago

    Allahabad, Uttar Pradesh, India beBeeCloudflare Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job DescriptionWe are seeking a seasoned Cloudflare WAF specialist to take on key responsibilities in ensuring the security and integrity of our websites.Manage Threats: Configure and manage Cloudflare WAF rules, page shields, and threat intelligence filters to provide optimal protection against web threats.Tune Policies: Optimize policies to minimize false...

  • Web Analytics Professional

    2 days ago

    Allahabad, Uttar Pradesh, India beBeeAnalytics Full time ₹ 15,00,000 - ₹ 20,00,000

    Web Analytics SpecialistAbout the Role:We are seeking a skilled and data-driven professional to lead the end-to-end tracking and analytics setup for our web platform.This role is responsible for designing and implementing a comprehensive measurement framework, ensuring accurate data collection, and delivering actionable insights that drive business...

  • Senior Web Development Specialist

    2 days ago

    Allahabad, Uttar Pradesh, India beBeeSoftware Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job DescriptionThe Senior Frontend Engineer will play a pivotal role in the development of high-performance web applications using Angular. Key responsibilities include designing, developing, and maintaining responsive web applications, collaborating with cross-functional teams to deliver seamless user experiences, integrating UI with relational databases...

  • Content Management Specialist

    1 week ago

    Allahabad, Uttar Pradesh, India beBeeContent Full time ₹ 13,50,000 - ₹ 20,25,000

    Job Title: Content Management SpecialistThe ideal candidate will be responsible for creating and maintaining testing checklists for content quality assurance, working closely with cross-functional teams, and utilizing AEM components, templates, and workflows to support content management processes.Key Responsibilities:Manage AEM authoring tasks, including...

  • Email Marketing Specialist

    2 days ago

    Allahabad, Uttar Pradesh, India AVingenious Technologies Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Company DescriptionAVingenious Technologies specializes in designing and developing websites that offer complete information solutions. They provide services such as web design, web development, SEO, and SMO. The company's mission is to deliver the highest quality services, ensure client satisfaction, and offer the best quality/price ratio in the industry....

  • Advanced Web Development Specialist

    6 days ago

    Allahabad, Uttar Pradesh, India beBeeSenior Full time ₹ 10,00,000 - ₹ 30,00,000

    Senior Frontend Engineer PositionJob Description:We are seeking an experienced React/Next.js developer with in-depth knowledge of React 18 and Next.js 14 to join our team.The ideal candidate will have a strong track record of delivering high-performance, real-time web applications in complex domains like trading or finance.Your primary responsibilities will...