Product Data Ingestion Specialist

1 week ago

Gandhinagar, Gujarat, India beBeeScraping Full time ₹ 1,80,00,000 - ₹ 2,00,00,000
Job Description

We are seeking a Senior Web Scraping Engineer to join our team at triplechoice-inc. This role is responsible for designing and implementing high-throughput product data ingestion pipelines across hundreds of domains.

The ideal candidate will own the crawling/extraction layer end-to-end, including HTTP-first crawling with a Playwright fallback, per-domain learned selectors, and reliable PDF handling. Additionally, they will drive automation around scheduling, retries, and monitoring to ensure runs are hands-off.

Key Responsibilities
  • Design an HTTP-first crawler with Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs for incremental runs.
  • Build a lightweight classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles/backoff and add URL normalization/canonicalization and de-dup.
  • Handle PDF discovery & download with deduplication and size/concurrency caps.
  • Apply Playwright browser automation resource budgets and integrate third-party APIs as first-class sources.
  • Own automation & orchestration for scheduled runs, idempotent retries, and alerting.
  • Create per-domain selectors with verification on hold-outs and re-learn only when health drops.
Must-Have Qualifications
  • 4+ years of Python experience, including 2+ years building production web crawlers at scale.
  • Strong expertise in Scrapy or aiohttp/asyncio and Playwright (or Puppeteer) in production.
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching.
Nice to Have
  • Go or Node.js experience for high-performance crawlers.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics.

Please apply with your resume and links to relevant repos or code samples. Include concise notes on a crawler you ran at 100+ sites/day, how you handled rate limits/retries, and your approach to PDF discovery/dedup.

  • Data Ingestion Specialist

    5 days ago

    Gandhinagar, Gujarat, India beBeeData Full time ₹ 45 - ₹ 55

    Job Title: Data Ingestion SpecialistImmediate Opportunity AvailableWe are seeking an experienced professional to design and implement data ingestion solutions. This role requires a strong understanding of data ingestion processes, focusing on integrating various data sources into Databricks.Key Responsibilities:Design and develop efficient data ingestion...

  • IoT Data Engineer Specialist

    1 week ago

    Gandhinagar, Gujarat, India beBeeIoT Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: IoT Data Engineer Specialist">We are seeking a highly skilled IoT Data Engineer to join our team.Key Responsibilities:Develop and maintain data engineering solutions leveraging AWS IoT services, utilizing Python programming to build, optimize, and automate data pipelines.Work with large-scale IoT data ingestion, processing, and storage...

  • Data Migration Specialist

    1 week ago

    Gandhinagar, Gujarat, India beBeeMigrations Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Overview:We're looking for a highly skilled Data Migration Specialist with expertise in migrating ETL pipelines and plugins from Fivetran into Nexla/Snowflake. The ideal candidate will design, build, and maintain automated ingestion and transformation processes to deliver high-quality data under tight deadlines.Key Responsibilities:Migrate data pipelines...

  • Data Architect Specialist

    1 week ago

    Gandhinagar, Gujarat, India beBeeDataSpecialist Full time ₹ 15,00,000 - ₹ 21,00,000

    About People Prime Worldwide:We are a global technology company, leveraging innovation to address complex business needs and create a sustainable future.Our team is committed to delivering exceptional results by designing, building, and maintaining robust data pipelines and scalable data solutions.Key Responsibilities:• Design and maintain robust data...

  • Data Infrastructure Specialist

    2 weeks ago

    Gandhinagar, Gujarat, India beBeeInfrastructure Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job DescriptionWe are seeking a highly skilled Data Infrastructure Specialist to join our team. The successful candidate will be responsible for designing, building, and operating scalable data infrastructure to support distributed computing and data orchestration for Large Language Model (LLM) research.The ideal candidate will have deep expertise in...

  • AI Innovation Specialist

    2 weeks ago

    Gandhinagar, Gujarat, India beBeeInnovation Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: AI Innovation SpecialistWe are seeking a skilled AI Innovation Specialist to support the development and deployment of advanced artificial intelligence solutions.The successful candidate will be responsible for implementing solutions that leverage Machine Learning, Natural Language Processing, and emerging technologies.This includes developing...

  • Big Data Innovation Lead

    1 week ago

    Gandhinagar, Gujarat, India beBeeDataAnalysis Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title:Business Intelligence Specialist\Key ResponsibilitiesDevelop and maintain large-scale data processing systems using Java, Scala, and Spark.Design and implement data pipelines using tools like Sqoop, Kafka, and HDFS.Work with query languages such as Oracle SQL, Hive SQL, Spark SQL, Impala, and HBase to analyze and process big data.Collaborate with...

  • Data Engineering Solutions Specialist

    1 week ago

    Gandhinagar, Gujarat, India beBeeDataEngineering Full time ₹ 18,66,000 - ₹ 25,18,000

    Job Description:We are seeking a highly skilled Data Engineering Solutions Specialist to join our team. In this role, you will be responsible for designing, developing, and managing data pipelines and transformation frameworks that enable analytics, reporting, and business intelligence capabilities across the enterprise.You will work closely with data...

  • Salesforce Data Cloud Specialist

    1 week ago

    Gandhinagar, Gujarat, India beBeeDataCloud Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Salesforce Developer – Data CloudThis is an exciting opportunity to work with a leading organization as a Salesforce Data Cloud professional.We are seeking a talented individual with strong experience in data modeling and data integration to design and implement Data Cloud solutions that unify customer data from multiple systems.Main...

  • Cloud Data Architect

    2 weeks ago

    Gandhinagar, Gujarat, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Seeking a skilled Cloud Data Engineer to design, develop and maintain complex data architectures using Databricks and related technologies. The ideal candidate will have experience in building scalable data products and working with big data platforms.Job DescriptionAs a Cloud Data Engineer, you will be responsible for analyzing and understanding existing...