Senior Data Ingestion Specialist

5 days ago

Udaipur, Rajasthan, India beBeeData Full time ₹ 50,00,000 - ₹ 1,00,00,000
Job Description

We are building a high-throughput product data ingestion pipeline across hundreds of domains, aiming to capture and process vast amounts of data with speed and accuracy.

About the Role:

You will own the crawling/extraction layer end-to-end, ensuring a seamless experience for users. This includes designing an HTTP-first crawler with a Playwright fallback for JS-heavy pages, implementing sitemap diffing and conditional GETs for incremental runs, and enforcing per-domain throttles/backoff to prevent overloading.

  1. Data Ingestion Specialist
  2. Crawler Development Expert
  3. Automation Engineer
Responsibilities:
  • Data Collection and Processing
  • Scraping and Extraction Layer Development
  • Automation and Orchestration
  • Error Handling and Monitoring
  • System Maintenance and Optimization

The ideal candidate will have 4+ years of experience in Python development, including 2+ years of building production web crawlers at scale. They should be proficient in Scrapy or aiohttp/asyncio and Playwright (or Puppeteer) in production, with hands-on experience in proxy management, polite anti-bot tactics, and per-domain rate limiting.

Required Skills and Qualifications
  • Python Programming
  • Web Crawling and Scraping
  • Ajax and API Integration
  • HTML Parsing and CSS/XPath
  • Database Management and Querying
Benefits

This role offers a challenging and rewarding opportunity to work on complex data ingestion pipelines, with opportunities for professional growth and development. The ideal candidate will be highly motivated, detail-oriented, and able to work effectively in a fast-paced environment.

Others

Please note that we are an equal opportunities employer and welcome applications from all qualified candidates. We look forward to hearing from you

  • Data Ingestion Specialist

    2 days ago

    Udaipur, Rajasthan, India beBeeIngestion Full time ₹ 16,00,000 - ₹ 22,00,000

    Data Ingestion SpecialistAs a Data Ingestion Specialist, you will play a pivotal role in our data management strategy by ensuring seamless integration of various data sources into Databricks.This position requires expertise in CI/CD and Kubernetes to design, develop, and optimize data ingestion pipelines for scalable workflows.Key responsibilities include...

  • Data Governance Specialist

    2 weeks ago

    Udaipur, Rajasthan, India beBeeDataGovernance Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Key Responsibilities of a Data Governance SpecialistMetadata Ingestion: Develop and implement data ingestion processes to populate Collibra's platform with metadata from various sources.Data Lineage Stitching: Utilize data lineage techniques to integrate datasets across systems, ensuring seamless data flow.Workflow Configuration and Customization: Configure...

  • Senior Data Infrastructure Specialist

    1 week ago

    Udaipur, Rajasthan, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Role Summary:We are seeking a highly skilled Data Infrastructure Specialist to join our organization. As a key member of our team, you will be responsible for designing and implementing scalable data infrastructure using cloud-native technologies.As a Senior Data Engineer at our organization, you will work closely with cross-functional teams to define data...

  • Senior Data Solutions Architect

    2 weeks ago

    Udaipur, Rajasthan, India beBeeAdobe Full time ₹ 20,00,000 - ₹ 25,00,000

    Job DescriptionOur organization seeks a highly skilled Senior Adobe Experience Platform (AEP) Developer to play a critical role in the Support & maintenance, design, development, and implementation of innovative customer data solutions within the AEP ecosystem. The ideal candidate will have a deep understanding of the AEP suite, Experience Data Model (XDM),...

  • # Senior Data Engineer – Cloud Integration Specialist

    2 weeks ago

    Udaipur, Rajasthan, India beBeeCloudIntegration Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Senior Data Engineer – Cloud Integration SpecialistWe are seeking a seasoned Senior Data Engineer to play a pivotal role in designing and implementing scalable data pipelines using Azure technologies.Key Responsibilities:Design, build, and maintain large-scale data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake...

  • Senior Data Specialist

    1 week ago

    Udaipur, Rajasthan, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Data EngineerWe're seeking a highly skilled data professional to drive the development and implementation of our ETL/ELT pipelines using Snowflake Tasks, Streams, and Stored Procedures.Key Responsibilities:Pipeline Development: Design and implement efficient data pipelines leveraging Snowflake's capabilities for improved query performance.Data Modeling:...

  • Senior Data Engineering Lead

    2 weeks ago

    Udaipur, Rajasthan, India beBeeDataEngineer Full time US$ 1,50,000 - US$ 2,00,000

    Job Title: Senior Data Engineer - SMEAbout the RoleWe are seeking a highly skilled Senior Data Engineer to design, build and maintain our modern data platform.This role is ideal for someone with deep technical expertise in ETL pipeline design, data modeling and data infrastructure who thrives in a fast-paced, engineering-driven environment.You will work...

  • Cybersecurity Implementation Specialist

    6 days ago

    Udaipur, Rajasthan, India beBeeImplementation Full time ₹ 1,20,00,000 - ₹ 1,50,00,000

    Cybersecurity Implementation SpecialistWe are seeking a highly skilled Cybersecurity Implementation Specialist to join our team.This role involves developing custom parsers to extract and normalize data from diverse sources, including logs, network traffic, and endpoint data. Additionally, the ideal candidate will have expertise in creating and maintaining...

  • Senior IoT Data Engineer

    6 days ago

    Udaipur, Rajasthan, India beBeeIoTEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Senior IoT Data EngineerWe are seeking a skilled Senior IoT Data Engineer to join our team.The ideal candidate will have 5-7 years of experience in designing and implementing scalable IoT data solutions using AWS IoT services.Proficiency in Python programming, with experience in building, optimizing, and automating data pipelines.Collaboration...

  • Data Engineering Specialist

    1 week ago

    Udaipur, Rajasthan, India beBeeDataEngineering Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Job Title: Data Engineering SpecialistJob Description:We are seeking a skilled data engineering professional to design, build, and maintain efficient data pipelines and workflows.Ensure high data quality and effective collaboration with cross-functional teams.Key Responsibilities:Design and implement scalable data processing systems.Develop reusable data...