High Throughput Data Engineer

2 weeks ago


Aurangabad, Maharashtra, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

Optimize Your Data Ingestion Pipeline with Our Senior Web Scraping Engineer.

We are seeking a seasoned expert to design and implement a high-throughput data ingestion pipeline across numerous domains. This is a key position in our organization, responsible for designing and implementing an HTTP-first crawler with a Playwright fallback for dynamic content pages.

The ideal candidate will own the crawling/extraction layer end-to-end, including sitemap diffing, conditional GETs, URL normalization, and de-duplication. Additionally, integrating vendor/public APIs wherever available to complement crawling and automate scheduling, retries, and monitoring is essential.

Strong skills in building production web crawlers at scale using Scrapy or aiohttp/asyncio are required. Proficiency in CSS/XPath parsing, proxy management, anti-bot tactics, rate limiting, and PDF handling is also necessary.

Automation/Orchestration expertise in Airflow/Temporal/Celery or equivalent schedulers/queues is essential for managing workflows and tasks efficiently. Knowledge of Docker, Linux basics, and clear communication skills are highly valued.

Our team aims to create a seamless experience for users by leveraging cutting-edge technology and innovative solutions. If you're passionate about data-driven insights and have a strong background in web scraping and automation, we encourage you to apply.



  • Aurangabad, Maharashtra, India beBeeData Full time ₹ 25,00,000 - ₹ 35,00,000

    Senior Data Engineer - AWS ExpertWe are seeking an experienced Senior Data Engineer with a strong background in AWS services to join our team. The ideal candidate should have a minimum of 2+ years of experience in designing, developing and implementing end-to-end data engineering solutions using AWS services such as Glue, EMR, Lambda and DynamoDB.Design and...

  • Master Data Architect

    2 weeks ago


    Aurangabad, Maharashtra, India beBeeDataEngineering Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Data Engineering ExpertWe're seeking a highly skilled data engineer to join our compact team. This is a critical role in the development of large-scale LLMs, requiring expertise in scalable data infrastructure and complex data processing.Job DescriptionThe ideal candidate will design, build, and operate fault-tolerant data infrastructure to support...


  • Aurangabad, Maharashtra, India beBeeBackend Full time ₹ 18,99,999 - ₹ 24,99,999

    Senior Backend Data EngineerThe organization seeks a seasoned Senior Backend Data Engineer to create high-quality data pipelines, integrate systems and interfaces, and ensure seamless integration with existing infrastructure.Key responsibilities include developing crawling/fetch pipelines using API-first approach, parsing/normalizing job postings & CVs,...


  • Aurangabad, Maharashtra, India beBeeSoftware Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Overview We are seeking a skilled software professional to design and build high-performance systems using modern technologies. You will collaborate with our team to create reliable infrastructure that supports rapid user growth. Design and develop scalable server-side applications using TypeScript and Nest.js on Node.js, focusing on high-performance,...

  • AI Data Specialist

    2 weeks ago


    Aurangabad, Maharashtra, India beBeeDataScientist Full time ₹ 17,03,615 - ₹ 21,14,537

    At aceolution, we are seeking highly skilled professionals to join our Data team in GenAI.Job Description:The successful candidate will collaborate with GEN AI researchers and engineers to understand data collection/evaluation requirements. They will develop detailed workflows to communicate effectively with a larger team, execute quickly on collection &...


  • Aurangabad, Maharashtra, India beBeeLeadership Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Technical Leadership OpportunityJob Summary:We are seeking a seasoned technical leader to drive engineering excellence and high-quality delivery. The ideal candidate will possess deep architectural expertise, hands-on technical skills, and proven track records in building and managing high-throughput systems.The role requires strong technical leadership with...

  • Data Engineer

    2 weeks ago


    Aurangabad, Maharashtra, India Insight Global Full time

    100% Remote Data Engineer Contract through the end of the year, high possibility of extensions*must be able to start a new role ASAP*- 5+ years of data engineer experience- Experience with AWS and Snowflake- Strong knowledge of the ETL processInsight Global is looking for a data engineer to join the analytics team at a global medical device client.This...


  • Aurangabad, Maharashtra, India beBeeData Full time US$ 1,20,000 - US$ 2,00,000

    Big Data Engineer PositionWe are seeking a skilled Big Data Engineer to design, build, and maintain high-performance data pipelines that process large datasets.Key Responsibilities:Design and implement scalable data processing systems.Develop and maintain efficient data pipelines.Collaborate with cross-functional teams to integrate data into business...


  • Aurangabad, Maharashtra, India beBeeBackend Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OverviewTo excel in this position, you will design and build microservices with low-latency architecture and high-throughput for real-time hotel search and distribution.Key Responsibilities:Create custom algorithms and data models to support pricing, availability, and inventory workflows.Own the development of AI-powered tools using Claude and Cursor to...


  • Aurangabad, Maharashtra, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 2,40,00,000

    Big Data Engineer RoleWe are looking for a skilled Big Data Engineer to join our team. The ideal candidate will have a strong background in data engineering, with experience in working with big data technologies such as Spark and Databricks.The successful candidate will be responsible for designing, implementing, and maintaining large-scale data systems....