High Performance Data Ingestion Engineer

1 week ago

Kota, Rajasthan, India beBeeScraping Full time ₹ 12,00,000 - ₹ 15,00,000
Job Title

Senior Web Crawling Specialist

About the Role

We are seeking a skilled Senior Web Crawling Specialist to join our team. As a key member of our data ingestion pipeline, you will be responsible for designing and implementing high-throughput web crawlers that can handle large volumes of data.

The ideal candidate will have a strong background in Python, Scrapy, and Playwright, with experience in building production-ready web crawlers at scale. You should also be familiar with ETag/Last-Modified, retries, backoff, and HTTP caching.

In addition to your technical skills, you should be able to communicate clearly and effectively, both technically and non-technically. This role requires strong ownership and a pragmatic approach to problem-solving.

Key Responsibilities
  • Design and implement high-throughput web crawlers using Scrapy, aiohttp, and Playwright.
  • Develop and maintain per-domain selectors using YAML, with verification on hold-outs.
  • Integrate third-party APIs as first-class sources, handling authentication, pagination, and rate limits.
  • Ship observability metrics, including field coverage, error rates, retries, and average page time.
Requirements
  • 4+ years of Python experience, including 2+ years building production web crawlers at scale.
  • Strong knowledge of Scrapy, aiohttp, and Playwright in production environments.
  • Practical experience with proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL (auth, pagination, backoff) and building small internal services (FastAPI or similar).
  • Automation/Orchestration: Airflow/Temporal/Celery (or equivalent schedulers/queues) for scheduled runs and monitoring.
  • PDF handling (requests/HEAD, hashing, size limits) and file integrity checks.
Nice to Have
  • Go or Node.js experience for high-performance crawlers.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics.
  • Workflow engines: Airflow/Temporal/Argo/Celery.
  • Document extraction: Textract/Tika/Camelot/Tabula.
  • Search/analytics: Elasticsearch/OpenSearch; warehousing (Snowflake/Postgres).

  • High Performance Data Architect

    2 weeks ago

    Kota, Rajasthan, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Data ArchitectWe are seeking a visionary Senior Data Architect to lead the design and implementation of scalable, high-performance data solutions that support Sales, Marketing, Manufacturing, Finance, and Engineering across a world-class organization.Achieve the successful integration of multiple data pipelines using Databricks and Azure Data...

  • High-Performance Web Crawler Specialist

    2 weeks ago

    Kota, Rajasthan, India beBeeDataIngestion Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Senior Data Ingestion EngineerWe are seeking a highly skilled Senior Data Ingestion Engineer to design and implement a high-throughput product data ingestion pipeline across hundreds of domains. This key member of our team will be responsible for crafting a robust and scalable solution that meets the demands of our business.Key Responsibilities:Design an...

  • Cybersecurity Expert

    2 weeks ago

    Kota, Rajasthan, India beBeeCybersecurity Full time ₹ 80,00,000 - ₹ 1,50,00,000

    As a seasoned cybersecurity professional, you will play a pivotal role in bolstering the effectiveness of our clients' security and digital operations.Job OverviewThis key position entails developing custom parsers to extract and normalize data from diverse sources, including logs, network traffic, and endpoint data.Key ResponsibilitiesDesign, develop, and...

  • High-Performance Data Architect

    2 weeks ago

    Kota, Rajasthan, India beBeeDataEngineer Full time US$ 1,20,000 - US$ 1,80,000

    Job Opportunity: Senior Data EngineerWe are seeking a highly skilled data engineer to build intelligence infrastructure that powers enterprise transformation in complex telecommunications and IT Asset Management environments.Key Responsibilities:Design and implement advanced ETL pipelines using DBT, SSIS, Informatica, or Talend.Develop high-performance...

  • GCP Data Engineer

    3 weeks ago

    Kota, Rajasthan, India CustomerLabs 1P Data OPs Full time

    Position Overview:"Yesterday is history, tomorrow is a mystery, but today is a gift. That's why we call itthe present." - Master OogwayJoin CustomerLabs' dynamic data team as a Data Engineer and play a pivotal role intransforming raw marketing data into actionable insights that power our digitalmarketing platform. As a key member of our data infrastructure...

  • High-Performance Engineering Role

    2 weeks ago

    Kota, Rajasthan, India beBeePerformance Full time ₹ 40,00,000 - ₹ 45,00,000

    Job Title: Performance Test EngineerA high-performance engineering role requiring 6+ years of experience.Job DescriptionWe are seeking a skilled Performance Test Engineer to join our team. As a key member of our digital solutions group, you will be responsible for developing and executing performance tests to ensure the delivery of high-quality...

  • High-Performance Software Engineer

    2 weeks ago

    Kota, Rajasthan, India beBeeSoftware Full time ₹ 15,00,000 - ₹ 20,00,000

    System ArchitectWe're seeking a highly skilled System Architect to spearhead the design and development of our internal CMS platform.This role requires 2+ years of hands-on experience with Python, with a strong focus on scalable system architecture.A solid understanding of relational databases like MySQL and PostgreSQL is essential for success in this...

  • Cloud Data Engineering Specialist

    1 week ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OpportunityWe are seeking a skilled professional to fill the position of Senior Data Engineer with expertise in GCP and ETL.This role involves working on a variety of projects, including data ingestion, processing, and storage. The ideal candidate will have experience with cloud-based technologies, particularly Google Cloud Platform (GCP), and a strong...

  • Data Engineer

    3 weeks ago

    Kota, Rajasthan, India R Systems Full time

    Job Title: Data EngineerContract Period: 12 MonthsLocation: Offshore candidates accepted (Singapore Based Company)Work Timing : 6.30 AM to 3.30 PM or 7.00 AM to 4.00 PM (IST - India timing)ExperienceMinimum 4+ years as a Data Engineer or similar role.(Please don't apply if less than 4 years exp in Data Engineer)Proven experience in Python, Spark, and PySpark...

  • High-Performance Data Engineer in Revenue Cycle Management

    2 weeks ago

    Kota, Rajasthan, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Lead Data Engineer in Healthcare Revenue Cycle ManagementWe are seeking an experienced Lead Data Engineer to join our technology team and lead the development of a self-service data platform for reporting and analytics. The ideal candidate will have 8+ years of experience in data engineering, with 3+ years in senior roles, and strong skills in PySpark,...