Senior Web Data Engineer

2 days ago

Ellore, Andhra Pradesh, India beBeeDataIngestion Full time ₹ 13,07,000 - ₹ 18,34,000
High-Throughput Product Data Ingestion Pipeline EngineerWe're building a high-throughput product data ingestion pipeline across hundreds of domains. This involves designing an HTTP-first crawler with a Playwright fallback for JavaScript-heavy pages, implementing sitemap diffing and conditional GETs for incremental runs, and enforcing per-domain throttles and backoff.Key Responsibilities Include:
  • Design and implement a robust HTTP-first crawler with a Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs (ETag/Last-Modified) for incremental runs.
  • Develop a lightweight 'needs JS?' classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles and backoff to prevent overloading.
  • Add URL normalization and de-duplication with respect to hash PDFs.
  • Handle PDF discovery and download with HEAD requests for deduplication, size/concurrency caps, and SHA-256 keys.
  • Apply Playwright browser automation resource budgets to block images/fonts/analytics and kill outliers by size/CPU/time.
  • Integrate third-party APIs as first-class sources: handle authentication, pagination, and rate limits; unify API + crawl outputs.
  • Own automation and orchestration for scheduled runs, idempotent retries, and alerting.
  • Create per-domain selectors with verification on hold-outs; re-learn only when health drops.
  • Ship observability: per-site field coverage, error rates, retries, avg page time, and PDF success.
  • Maintain allow/deny paths; adhere to robots.txt and Terms of Service.
Ambitious candidates should have experience in web scraping, data ingestion, and automation. A strong understanding of HTTP protocols, web crawlers, and data processing is required. Additionally, proficiency in languages such as Python, JavaScript, or Java is necessary for this role.This position offers the opportunity to work on a high-impact project that drives business growth through data-driven decision making. We value innovation, collaboration, and continuous learning. If you're passionate about data engineering and want to be part of a dynamic team, apply now

  • Senior Web Developer

    4 days ago

    Ellore, Andhra Pradesh, India beBeeDeveloper Full time ₹ 5,00,000 - ₹ 10,00,000

    Role OverviewJob SummaryWe are seeking a seasoned web developer to design and implement scalable, efficient, and secure web applications using the Laravel framework.Main ResponsibilitiesDevelop high-quality web applications leveraging the full potential of the Laravel framework.Collaborate with cross-functional teams to understand project requirements,...

  • Senior Web Scraping Developer

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeWebScraping Full time ₹ 6,00,000 - ₹ 10,00,000

    Senior Web Scraping Developer Position">We are seeking an experienced Senior Python developer to design, develop and optimize large-scale web scraping solutions that power data-driven decision-making. The ideal candidate will possess proven expertise in Scrapy (mandatory), as well as strong skills in web scraping and automation.They will be responsible for...

  • Senior Data Engineer

    6 days ago

    Ellore, Andhra Pradesh, India beBeeMachineLearningEngineer Full time ₹ 1,92,00,000 - ₹ 2,56,00,000

    As a Senior Data Engineer, you will play a crucial role in enhancing our machine learning capabilities and credit risk management processes.Key Responsibilities:Design and implement end-to-end data engineering architecture, including model deployment and production maintenance.Develop and maintain scalable data models using Python, PySpark, NumPy, Pandas,...

  • Strategic Web Engineering Lead

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Strategic Web Engineering Lead As a senior web development leader, you will oversee the creation and deployment of cutting-edge web applications. Your key responsibilities include feature ownership from conception to delivery, encompassing architecture, implementation, testing, deployment, and optimization. The ideal candidate for this role possesses...

  • Senior Data Engineer

    2 days ago

    Ellore, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Senior Data EngineerAbout the RoleWe are seeking a highly skilled data engineer to join our team for an exciting in-house project. The ideal candidate will have extensive experience in ETL development and a strong background in Python, SQL, and automation.Key Responsibilities:Design and implement complex data pipelines using Python and SQL.Develop...

  • Web Development Expert

    4 hours ago

    Ellore, Andhra Pradesh, India beBeeEngineering Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Senior Web Engineering PositionAbout This RoleWe are seeking a skilled web engineer to join our team and contribute to the development of our next-generation recruitment and freelancing solutions.This position demands technical expertise, creative problem-solving, and high-quality code delivery.

  • Senior Data Engineer

    1 week ago

    Ellore, Andhra Pradesh, India beBeeData Full time ₹ 1,34,40,000 - ₹ 2,01,20,000

    Job Summary: We are seeking an experienced Senior Software Engineer to join our team. As a key member of the technical staff, you will be responsible for designing and implementing scalable data architectures, developing and optimizing ETL/ELT pipelines, and collaborating with stakeholders to ensure data availability.Key Responsibilities:Design and implement...

  • Senior Data Engineer

    6 days ago

    Ellore, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 19,00,000

    About this Role:A senior data engineer is responsible for designing, developing and maintaining large-scale data processing systems. They use their expertise in Airflow and Python to drive business growth.Collaborating with cross-functional teams ensures timely and accurate data delivery.Developing, implementing and monitoring data workflows helps leverage...

  • Core Software Engineer for Scalable Web Applications

    2 weeks ago

    Ellore, Andhra Pradesh, India beBeeBackendDeveloper Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job Title: Java Backend DeveloperWe are seeking a skilled Java Backend Developer to join our team at People Prime Worldwide. This role involves serving as a core member of an agile team that drives user story analysis and elaboration designs and develops responsive web applications using the best engineering practices.Serving as a core member of an agile...

  • Web Application Developer

    2 days ago

    Ellore, Andhra Pradesh, India beBeeSoftware Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Frontend EngineerAs a highly skilled Senior Frontend Engineer, you will play a key role in building responsive and user-centric web applications using Next.js, React, and Tailwind CSS.This position is ideal for someone who is not only technically strong but also demonstrates leadership potential. Individuals who take initiative and show impact will...