Data Ingestion Pipeline Manager

5 days ago

Alwar, Rajasthan, India beBeeExpert Full time US$ 70,000 - US$ 1,20,000
Product Data Ingestion Specialist
We are building a high-throughput product data ingestion pipeline across hundreds of domains.
You will own the crawling/extraction layer end-to-end, leveraging HTTP-first crawling with a Playwright fallback and per-domain learned selectors. Reliable PDF handling for datasheets/specs is also a key aspect of this role.
This role spans crawling (discovering & fetching pages via sitemaps/robots) and scraping (extracting structured specs, images, and PDFs into our schema). Key responsibilities include:
  • Design an HTTP-first crawler using Scrapy or aiohttp with a Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs with ETag/Last-Modified headers for incremental runs.

Key requirements for success in this position include expertise in web scraping, experience with Python libraries such as Scrapy and Playwright, and a strong understanding of crawling and extraction concepts.

  • Data Ingestion Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeIngestion Full time ₹ 12,00,000 - ₹ 20,00,000

    Job Title: Data Ingestion SpecialistAbout the Role:We are seeking an experienced Data Ingestion Specialist to join our team. The ideal candidate will have hands-on experience with data pipelines, automation, and infrastructure management to support scalable and efficient ingestion workflows.Main Responsibilities:Data Ingestion Pipelines: Design, develop, and...

  • Highly skilled Data Engineer required to drive data pipeline scalability and efficiency

    2 weeks ago

    Alwar, Rajasthan, India beBeeDataEngineering Full time ₹ 25,00,000 - ₹ 35,00,000

    Data Engineering Position">Design, develop and maintain scalable data pipelines for large-scale data ingestion, transformation and storage.Work with Big Data ecosystems (Hadoop, Spark, Hive or similar) to process structured and unstructured data efficiently.Develop efficient ETL processes to integrate data from multiple sources into data...

  • Data Pipeline Specialist

    1 week ago

    Alwar, Rajasthan, India beBeeDataProfessional Full time ₹ 18,00,000 - ₹ 25,00,000

    We are seeking a skilled Data Professional to develop and refine data pipelines, ensuring clean, accessible, and secure data for analytics and operations.Key Responsibilities:Create impactful business intelligence reports using BI tools such as Bold BI, Power BI, or SupersetDesign and implement robust data architectures using SQL Server, ClickHouse, and...

  • Data Engineering Expert

    1 week ago

    Alwar, Rajasthan, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    Unlock Data Engineering ExcellenceThe ideal candidate will be responsible for designing, developing, and managing data pipelines and transformation frameworks that enable analytics, reporting, and business intelligence capabilities across the enterprise.Key Responsibilities:Data Pipeline Development: Design and build scalable ETL/ELT pipelines using Azure...

  • Chief Data Solutions Architect

    1 week ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Senior Data Engineer RoleDesign and build scalable data pipelines on the cloud. This position involves working with Python, Scala, Apache Spark for big data processing on AWS.Implement data ingestion, transformation, and integration solutions using Spark, Glue, and SQL.Manage and optimize cloud storage and compute environments.Experience in Terraform and...

  • Senior Data Engineer(Lead)

    2 weeks ago

    Alwar, Rajasthan, India ITC Infotech Full time

    ResponsibilitiesAround 6+ years of experience in data engineering or cloud computing data developmentDesign, develop, and optimize ETL/ELT data pipelines using Apache Spark (PySpark or Scala), AWS Glue, and Azure Data FactoryWork with structured and unstructured data to build scalable ingestion and transformation workflows across cloud platforms.Build data...

  • Senior Cloud Data Solutions Developer

    1 week ago

    Alwar, Rajasthan, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Data EngineerWe are seeking an experienced Cloud Data Engineer to design, develop, and optimize robust data ingestion pipelines using GCP services. The ideal candidate will be proficient in GCP services like Pub/Sub, Dataflow, and BigQuery, with a foundational understanding of AI/ML workflows using Vertex AI.Key Responsibilities:Design and develop...

  • Data Architect

    2 weeks ago

    Alwar, Rajasthan, India beBeeBigDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    As a highly skilled professional, you will be responsible for designing and building robust data pipelines that handle high-volume financial data.Key Responsibilities:Design, develop, and manage end-to-end data pipelines for stocks, crypto, and other financial datasets.Integrate third-party APIs and data feeds into internal systems.Build and optimize data...

  • Data Engineering Specialist

    2 weeks ago

    Alwar, Rajasthan, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job TitleWe are seeking a highly skilled Data Engineering Specialist to lead the design, development and implementation of our data pipeline and migration requirements.Key Responsibilities:Ownership of Data PipelinesAutomated Ingestion and TransformationProcess ImprovementsTroubleshooting and RemediationComprehensive Knowledge of Functional and Technical...

  • Transforming Data Pipelines Specialist

    1 week ago

    Alwar, Rajasthan, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Data Engineering RoleA key responsibility of this role is to design and implement data pipelines, orchestrate workflows, and process large datasets.