High Performance Data Engineer

19 hours ago

Anand, Gujarat, India beBeeSoftware Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

Senior Web Scraping Engineer Job Description

">

About this Role:

  • Design and implement a high-throughput data ingestion pipeline across hundreds of domains.
  • Owning the crawling/extraction layer end-to-end, including HTTP-first crawling with Playwright fallback, per-domain learned selectors, and reliable PDF handling.

This role spans crawling (discovering & fetching pages via sitemaps/robots) and scraping (extracting structured specs, images, and PDFs into our schema).

  • Implement an HTTP-first crawler with Scrapy or aiohttp and a Playwright fallback for JS-heavy pages.
  • Build a lightweight classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles/backoff and add URL normalization/canonicalization and de-duplication.
  • Integrate third-party APIs as first-class sources and handle auth, pagination, and rate limits.
  • Maintain allow/deny paths and adhere to robots.txt and Terms of Service.
  • Containerize workers and collaborate with the data team on schemas/normalization.

You will work on building observability, containerizing workers, providing runbooks/CI, collaborating with the data team on schemas/normalization, maintaining allow/deny paths, adhering to robots.txt and Terms of Service.

We are looking for someone with 4+ years of Python experience, including 2+ years of building production web crawlers at scale. You should be strong with Scrapy or aiohttp/asyncio and Playwright, have practical proxy management and polite anti-bot tactics, and hands-on experience with ETag/Last-Modified, retries, backoff, and HTTP caching.

  • High-Performance Data Architect

    2 weeks ago

    Anand, Gujarat, India beBeeData Full time ₹ 14,12,450 - ₹ 25,48,245

    Big Data Engineer Opportunity We are looking for a skilled Big Data Engineer to join our team. As a key member of our data infrastructure group, you will be responsible for designing and building high-performance data pipelines that process massive amounts of data. Key Responsibilities: Design and implement scalable batch processing systems using Python and...

  • High-Performance Data Architect

    5 days ago

    Anand, Gujarat, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 30,00,000

    Senior Data Engineer RoleWe are seeking a highly skilled Senior Data Engineer to join our team.This role involves designing, building, and maintaining high-performance data pipelines to support consulting and analytics solutions. The successful candidate will work closely with cross-functional teams to develop and implement data models, ETL processes, and...

  • High-Performance Software Engineering Leader

    2 weeks ago

    Anand, Gujarat, India beBeeEngineering Full time US$ 18,00,000 - US$ 24,00,000

    Job OverviewWe are seeking a seasoned Senior Software Engineering Manager to lead the development of our platform for large language models. As a key member of our team, you will be responsible for designing, building, and scaling production applications that meet high standards of quality and performance.The ideal candidate will have a strong background in...

  • High-Performing Test Engineer

    5 days ago

    Anand, Gujarat, India beBeePerformance Full time ₹ 10,00,000 - ₹ 15,00,000

    About our workWe specialize in delivering high-quality product engineering services. Our team is focused on leveraging modern technologies to drive business success.Job Summary:Main Responsibilities:Gather and understand performance testing requirementsCreate load generation scripts using JMeterApply customizations as requiredSetup monitoring counters and...

  • High-Performance Engineer

    1 week ago

    Anand, Gujarat, India beBeePerformance Full time ₹ 10,08,000 - ₹ 1,55,25,200

    Job OverviewAbout this role: We are seeking a seasoned performance engineer to join our team. In this key position, you will be responsible for designing and implementing performance benchmarks, collaborating with cross-functional teams to enhance system performance and scalability, and leveraging profiling tools to identify bottlenecks.Key...

  • High-Performance Data Solutions Expert

    1 week ago

    Anand, Gujarat, India beBeeDataEngineer Full time ₹ 17,06,431 - ₹ 24,58,244

    Job Title: Data Engineer - Oil & Gas DomainWe are seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines and ETL/ELT solutions using AWS cloud services.The ideal candidate will have expertise in ingesting, processing, and storing structured/unstructured data from multiple sources (IoT devices, sensors, ERP systems, drilling...

  • High Performance Backend Software Engineer

    1 week ago

    Anand, Gujarat, India beBeeBackend Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Senior Backend EngineerAbout the RoleWe are seeking a highly skilled Senior Backend Engineer to contribute to our engineering team. The ideal candidate will have in-depth knowledge of Golang and experience with Ruby, with a strong focus on designing scalable, high-performance backend systems.Key Responsibilities:Design, build, and maintain large-scale...

  • High-Performance Software Engineer

    1 week ago

    Anand, Gujarat, India beBeeSoftwareEngineer Full time ₹ 90,00,000 - ₹ 1,50,00,000

    **Job Summary**We are seeking an experienced software engineer to lead the design, development and deployment of Oracle EPM Cloud solutions.Your key responsibilities will include designing and configuring Oracle PCS workflows to automate business processes, building and maintaining database objects in Oracle PL/SQL, and collaborating with finance teams to...

  • High Performance Backend Developer

    5 days ago

    Anand, Gujarat, India beBeeBackend Full time ₹ 18,00,000 - ₹ 21,00,000

    About UsWe are a fast-growing eSIM services platform that simplifies connectivity with powerful APIs, robust B2B and B2C interfaces, and seamless integrations. Our platform enables global eSIM lifecycle management, user onboarding, secure payment systems, and scalable deployments.Key ResponsibilitiesAPI Development: We design, develop, optimize, and maintain...

  • High Performance System Developer

    2 days ago

    Anand, Gujarat, India beBeeDeveloper Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job OverviewWe are seeking an expert in building high-performance systems to design and implement scalable server-side applications using cutting-edge technologies. You will work with a range of programming languages, frameworks, and databases to create reliable infrastructure that supports rapid user growth.Key Responsibilities:System Design and...