Web Crawling Expert

2 weeks ago

Kolkata, West Bengal, India beBeeSenior Full time ₹ 30,00,000 - ₹ 40,00,000

Job Title: Senior Web Scraping Engineer

About the Role

We are seeking a skilled Senior Web Scraping Engineer to join our team. As a key member of our engineering team, you will be responsible for designing and implementing a high-throughput product data ingestion pipeline across hundreds of domains.

Key Responsibilities:

  • Design an HTTP-first crawler with Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs for incremental runs.
  • Build a lightweight classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles/backoff and add URL normalization/canonicalization.
  • Handle PDF discovery & download with size/concurrency caps and SHA-256 keys.
  • Apply Playwright browser automation resource budgets and integrate third-party APIs as first-class sources.
  • Own automation & orchestration for scheduled runs, idempotent retries, and alerting.
  • Ship observability metrics and maintain allow/deny paths.
Requirements

To be successful in this role, you will need:

  • 4+ years Python experience with 2+ years building production web crawlers at scale.
  • Strong expertise in Scrapy or aiohttp/asyncio and Playwright (or Puppeteer).
  • Practical proxy management and polite anti-bot tactics.
  • Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery for scheduled runs and monitoring.
Preferred Qualifications

Prior experience with:

  • Go or Node.js for high-performance crawlers.
  • Cloud infrastructure: AWS/GCP, S3, ECS/Kubernetes; IaC basics.
  • Workflow engines: Airflow/Temporal/Argo/Celery.

Competitive compensation includes salary and benefits.

  • Expert Web Application Developer

    2 weeks ago

    Kolkata, West Bengal, India beBeeDeveloper Full time ₹ 15,00,000 - ₹ 25,00,000

    Web Development ExpertWe are seeking a highly skilled and experienced Web Development Expert to join our team. The ideal candidate will have strong hands-on expertise in Angular, .NET Core, SQL Server, and Selenium-based test automation.The successful candidate will be responsible for designing, developing, and maintaining scalable web applications while...

  • Web Development Expert

    2 weeks ago

    Kolkata, West Bengal, India beBeeAgentic Full time ₹ 20,00,000 - ₹ 25,00,000

    Web Development Expert In this pivotal role, you will spearhead the creation of cutting-edge front-end applications using modern frameworks like React, Angular, and Vue.js. You will be responsible for translating user experience requirements into responsive, reusable, and high-performance interfaces that captivate users. Your key objectives include...

  • PHP Web Development Expert

    2 weeks ago

    Kolkata, West Bengal, India beBeeLaravel Full time ₹ 12,00,000 - ₹ 15,00,000

    Job Opening:We are seeking an experienced PHP developer to join our dynamic team.As a PHP developer, you will be responsible for developing and maintaining high-quality web applications using the Laravel framework.You will collaborate with cross-functional teams to understand project requirements, design and implement solutions, and ensure the smooth...

  • Web Development Expert

    2 weeks ago

    Kolkata, West Bengal, India beBeeDeveloper Full time ₹ 15,000 - ₹ 24,000

    Job Opportunities:Position SummaryWe are seeking a skilled Web Developer to join our team. The ideal candidate will have hands-on experience in WordPress development, with strong knowledge of programming languages such as PHP, HTML, CSS, and JavaScript.Key ResponsibilitiesDevelop and maintain web applications using WordPress platforms.Create customized user...

  • Web Development Enthusiast Wanted

    2 weeks ago

    Kolkata, West Bengal, India beBeeWebDevelopment Full time ₹ 8,00,000 - ₹ 10,00,000

    Unlock Your Web Development PotentialWe're excited to offer a unique opportunity for passionate individuals who want to gain hands-on experience with cutting-edge technologies.As a skilled web development enthusiast, you'll work on real-world projects and collaborate with our team of experts. You'll have the chance to:Gain practical experience with React,...

  • Senior Web Development Specialist

    2 weeks ago

    Kolkata, West Bengal, India beBeeEcommerce Full time ₹ 8,00,000 - ₹ 12,00,000

    Expert E-commerce Solution ProviderCreatikartta is dedicated to delivering innovative marketing solutions that bridge the gap between marketing and sales. Our team excels in harnessing the synergy between these two crucial aspects of business operations.ResponsibilitiesWe develop, design, and maintain client websites on Wix and Shopify platforms.Our skilled...

  • Full Stack Web Application Expert

    1 week ago

    Kolkata, West Bengal, India beBeeBackendDeveloper Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    **Senior Web Application Developer Position OverviewThis is a full-time, remote role for a skilled web application developer. The successful candidate will be responsible for designing, developing, and maintaining high-quality web applications. Key responsibilities include writing clean, scalable code, collaborating with cross-functional teams,...

  • Web Solutions Architect

    2 weeks ago

    Kolkata, West Bengal, India beBeeDevelopment Full time ₹ 9,00,000 - ₹ 15,00,000

    We are seeking a skilled Web Solutions ArchitectA Web Solutions Architect is required to design and develop high-quality, conversion-focused websites using Elementor and WordPress.The ideal candidate will have expertise in building responsive, pixel-perfect pages, developing custom WordPress functionality with PHP and JavaScript, writing clean HTML and CSS,...

  • Web Performance Expert

    1 week ago

    Kolkata, West Bengal, India beBeeDigitalGrowth Full time ₹ 18,00,000 - ₹ 22,00,000

    Website Optimization SpecialistThis role involves driving digital growth and conversions by implementing updates, optimizing conversion rates and site performance.You will be responsible for maintaining and improving a custom WordPress website, managing content changes, and providing web analytics insights that support lead generation initiatives.Implement...

  • Software Engineering Expert

    2 weeks ago

    Kolkata, West Bengal, India beBeeDevelopment Full time ₹ 15,00,000 - ₹ 22,00,000

    Job Summary:We are seeking a highly skilled Software Engineering Expert to join our team. As an Application Developer, you will design and build software applications that meet business requirements.Main Responsibilities:Collaborate with cross-functional teams to understand their needs and develop innovative solutionsDesign, build, and configure software...