Expert Crawler Developer

2 days ago

Meerut, Uttar Pradesh, India beBeeWebScraping Full time ₹ 15,00,000 - ₹ 25,00,000
Job Title: Senior Web Scraping Engineer

We are seeking a seasoned Senior Web Scraping Engineer to own the crawling/extraction layer end-to-end.

This role spans crawling (discovering & fetching pages via sitemaps/robots) and scraping (extracting structured specs, images, and PDFs into our schema).

  • Design an HTTP-first crawler with Scrapy or aiohttp, and Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs for incremental runs.
  • Build a lightweight classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles/backoff and URL normalization/canonicalization.
  • Handle PDF discovery & download and apply Playwright browser automation resource budgets.
  • Integrate third-party APIs as first-class sources and own automation & orchestration.
  • Ship observability metrics and maintain allow/deny paths.
Requirements
  • 4+ years Python experience, including 2+ years building production web crawlers at scale.
  • Strong skills in Scrapy or aiohttp/asyncio and Playwright.
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery for scheduled runs and monitoring.

Clear, pragmatic communication and strong ownership are key qualifications for success in this role.

  • Data Driven Web Crawler Specialist

    6 days ago

    Meerut, Uttar Pradesh, India beBeeWebCrawler Full time ₹ 8,00,000 - ₹ 12,00,000

    As a Junior Web Crawling Engineer, you will be responsible for designing and implementing web crawlers to extract valuable insights from the web. You will work on building and maintaining complex web scraping projects, utilizing Python-based tools and frameworks.Key Responsibilities:Maintain and enhance existing web scraping and data crawling...

  • Software Development Expert

    6 days ago

    Meerut, Uttar Pradesh, India beBeeSoftwareDevelopment Full time ₹ 8,00,000 - ₹ 15,00,000

    Software Development ExpertWe are seeking a highly skilled software development expert to fill this key position. The ideal candidate will possess a strong background in software development, with a focus on creating efficient, scalable, and reliable systems.

  • Cloud Software Development Expert

    1 week ago

    Meerut, Uttar Pradesh, India beBeecloud Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Cloud Software Development ExpertWe are seeking an experienced Cloud Software Development Expert to contribute to all aspects of software development, from architecture to design, estimation, development, testing, and release.

  • AI Development Expert

    3 days ago

    Meerut, Uttar Pradesh, India beBeeAI Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Job Title: AI Development ExpertSeeking a skilled professional to design and deliver AI-powered solutions using GitHub Copilot and Amazon Q.Develop and implement AI-driven projects, leveraging expertise in generative AI tools and technologies.Mentor and guide teams to ensure successful project outcomes.Collaborate with cross-functional teams to integrate AI...

  • Software Development Expert

    6 days ago

    Meerut, Uttar Pradesh, India beBeeSoftware Full time ₹ 90,00,000 - ₹ 1,20,00,000

    Job Title: Software Development ExpertAbout This Role:We are seeking an experienced software development expert to join our team. As a key member of our software development team, you will be responsible for delivering end-to-end technology solutions to clients worldwide.Key Responsibilities Include:Requirement gathering and designDevelopment of software...

  • Expert Integration Developer

    9 hours ago

    Meerut, Uttar Pradesh, India beBeeIntegration Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Integration Expert WantedWe are looking for a seasoned integration expert to join our team.This role involves designing and implementing scalable solutions using WSO2, microservices, and Java/Spring Boot, while driving API governance and integration best practices.Key Responsibilities:Lead integration solutions using WSO2 API Manager and Enterprise/Micro...

  • Expert iSeries Developer

    47 minutes ago

    Meerut, Uttar Pradesh, India beBeeRPG Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Senior Developer Position for iSeries ExpertSeeking a talented individual with extensive experience in the iSeries platform and RPGLE programming.Strong expertise in Integrated Language Environment (ILE) is essential.Experienced in creating, debugging, and optimizing RPG/CL programs.Ability to design and implement complex sub-procedures, subroutines, and...

  • Backend Developer Expert

    4 days ago

    Meerut, Uttar Pradesh, India beBeeExpert Full time ₹ 15,00,000 - ₹ 20,10,000

    Backend Developer ExpertWe are seeking a highly skilled Backend Developer to work on serverless applications using NodeJS/TypeScript.Design and implement scalable, secure backend services using AWS technologies.Create robust, maintainable code with Vitest for efficient testing.Collaborate with cross-functional teams to deliver high-quality features.Ensure...

  • Expert MuleSoft Software Developer

    6 days ago

    Meerut, Uttar Pradesh, India beBeeIntegration Full time ₹ 10,00,000 - ₹ 20,00,000

    Software Integration Expert WantedWe are seeking an experienced Software Integration Expert to work with MuleSoft integration, API, and Anypoint Platform. Our ideal candidate is a skilled professional who can leverage their knowledge of software integration to drive business success.Required Skills:Proficiency in MuleSoft integrationExperience with Anypoint...

  • Enterprise Software Development Expert

    2 weeks ago

    Meerut, Uttar Pradesh, India beBeeSoftwareDevelopment Full time US$ 1,20,000 - US$ 1,40,000

    Enterprise Software Development ExpertAre you a seasoned business analyst seeking a new challenge? We are looking for a skilled Enterprise Software Development Expert to join our team. In this role, you will be responsible for gathering and defining business requirements, analyzing business processes, and ensuring successful iteration management.Key...