High-Performance Web Crawler Specialist

2 weeks ago

Kota, Rajasthan, India beBeeDataIngestion Full time ₹ 1,80,00,000 - ₹ 2,50,00,000
Senior Data Ingestion Engineer

We are seeking a highly skilled Senior Data Ingestion Engineer to design and implement a high-throughput product data ingestion pipeline across hundreds of domains. This key member of our team will be responsible for crafting a robust and scalable solution that meets the demands of our business.

Key Responsibilities:

  • Design an HTTP-first crawler with Playwright fallback for JS-heavy pages, ensuring seamless navigation through complex web applications.
  • Implement sitemap diffing and conditional GETs for incremental runs, guaranteeing efficient and accurate data retrieval.
  • Build a lightweight 'needs JS?' classifier to auto-route HTTP vs Playwright, optimizing data collection and processing workflows.
  • Enforce per-domain throttles/backoff and add URL normalization/canonicalization and de-duplication, minimizing errors and maximizing data quality.
  • Handle PDF discovery & download, apply Playwright browser automation resource budgets, and integrate third-party APIs, streamlining data extraction and processing.
  • Own automation & orchestration for scheduled runs, idempotent retries, and alerting, ensuring continuous data flow and minimal downtime.
  • Ship observability: per-site field coverage, error rates, retries, avg page time, and PDF success, providing actionable insights into data collection and processing performance.
  • Maintain allow/deny paths, adhere to robots.txt and Terms of Service, respecting web protocols and regulations.

Requirements:

  • 4+ years Python experience, including 2+ years building production web crawlers at scale, with a strong understanding of Scrapy or aiohttp/asyncio and Playwright (or Puppeteer).
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting, ensuring secure and compliant data collection.
  • Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching, optimizing data retrieval and processing efficiency.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing, extracting valuable insights from complex web structures.
  • APIs: consuming REST/GraphQL (auth, pagination, backoff) and building small internal services, integrating data sources and creating streamlined workflows.
  • Automation/Orchestration: Airflow/Temporal/Celery (or equivalent schedulers/queues), managing data flows and ensuring seamless execution.
  • PDF handling and file integrity checks, verifying data accuracy and consistency.
  • Queues (Redis/Kafka), Docker, Linux basics; comfort with logs/metrics, monitoring system performance and identifying areas for improvement.

Nice to Have:

  • Go or Node.js experience for high-performance crawlers, leveraging language-specific features for enhanced efficiency.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics, deploying and managing scalable infrastructure.
  • Workflow engines: Airflow/Temporal/Argo/Celery, automating complex data processing pipelines.
  • Document extraction: Textract/Tika/Camelot/Tabula, unlocking valuable insights from unstructured data.
  • Search/analytics: Elasticsearch/OpenSearch; warehousing (Snowflake/Postgres), providing actionable intelligence and informing business decisions.
  • LLM-assisted selector generation with deterministic verification, harnessing AI-driven insights for improved data collection.

  • High Performance Data Ingestion Engineer

    1 week ago

    Kota, Rajasthan, India beBeeScraping Full time ₹ 12,00,000 - ₹ 15,00,000

    Job TitleSenior Web Crawling SpecialistAbout the RoleWe are seeking a skilled Senior Web Crawling Specialist to join our team. As a key member of our data ingestion pipeline, you will be responsible for designing and implementing high-throughput web crawlers that can handle large volumes of data.The ideal candidate will have a strong background in Python,...

  • Expert Web Data Extraction Specialist

    2 weeks ago

    Kota, Rajasthan, India beBeeDataExtraction Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Description:We are seeking an Expert Web Data Extraction Specialist to join our team. The ideal candidate will be responsible for designing and implementing web crawlers, extracting valuable insights from the web, and ensuring data quality.The successful candidate will have strong Python programming skills and experience in web scraping frameworks,...

  • High-Performing Systems Specialist

    2 weeks ago

    Kota, Rajasthan, India beBeeperformance Full time US$ 4,00,000 - US$ 8,00,000

    Job Title: Performance Optimization SpecialistRole Overview:We are seeking a skilled Performance Optimization Specialist to join our team. As a key member of our organization, you will play a critical role in ensuring the high availability and performance of our systems.Main Responsibilities:Key Job Functions:Design, develop, and implement performance...

  • Senior High Performance Test Specialist

    2 weeks ago

    Kota, Rajasthan, India beBeePerfomance Full time ₹ 9,00,000 - ₹ 12,15,000

    High-Performance Testing SpecialistAbout This RoleThis is an exciting opportunity to join our engineering team as a High-Performance Testing Specialist.Job DescriptionWe are seeking a skilled and experienced Performance Tester to play a crucial role in developing a comprehensive performance testing strategy. The successful candidate will be responsible for...

  • High-Performance Testing Specialist

    2 weeks ago

    Kota, Rajasthan, India beBeePerformance Full time ₹ 9,00,000 - ₹ 15,00,000

    Job Title: High-Performance Testing SpecialistAs a key member of our software development team, you will be responsible for ensuring the optimal performance and efficiency of our applications.You will design and execute comprehensive performance tests using industry-leading tools like JMeter, Dynatrace, and AppDynamics to identify areas for improvement and...

  • High-Performance Frontend Developer

    2 weeks ago

    Kota, Rajasthan, India beBeeSoftwareEngineer Full time ₹ 50,00,000 - ₹ 55,00,000

    As a Software Engineer on our Search Team, you will play a pivotal role in enabling users to quickly find and get started with APIs. With the platform growing rapidly, this translates into an ever-increasing volume of data created and consumed by users across teams.We are looking for a skilled professional who thrives on building seamless high-performance...

  • High-Powered Performance Tester

    2 weeks ago

    Kota, Rajasthan, India beBeePerformance Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title:Performance Test SpecialistKey Responsibilities:Design and execute performance tests to ensure application reliability and scalability.Collaborate with cross-functional teams to identify performance bottlenecks and develop optimization strategies.Develop and implement performance testing frameworks and automation scripts.Required Skills:5-9 years...

  • High-Performance Software Developer

    2 weeks ago

    Kota, Rajasthan, India beBeeSoftwareDeveloper Full time ₹ 20,00,000 - ₹ 25,00,000

    About our teamWe're a leading technology provider with over 4,800 employees and ~22,000 customers worldwide. Founded in 1999, we deliver a comprehensive event marketing and management platform for marketers and event professionals.Our company culture emphasizes innovation and empowerment, fostering an environment where individual contributors can think and...

  • High-Performance Testing Expert

    2 weeks ago

    Kota, Rajasthan, India beBeePerformanceTesting Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job DescriptionWe are looking for a skilled Performance Testing Specialist to join our team. The ideal candidate will have a strong background in performance testing, with experience using LoadRunner and APM tools like AppDynamics.The successful candidate will be responsible for designing, executing, and analyzing performance tests to ensure the optimal...

  • High-Performance Database Specialist

    2 weeks ago

    Kota, Rajasthan, India beBeeDataInfrastructure Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Infrastructure SpecialistAbout the Role:We seek a seasoned professional to oversee and maintain our mission-critical databases. The ideal candidate will play a key role in upgrading, migrating, and ensuring performance, availability, security, and scalability of our data infrastructure.Key Responsibilities:Lead and execute SQL Server upgrade...