Senior Product Data Specialist

2 days ago

Nellore, Andhra Pradesh, India beBeeWebScrapingEngineer Full time ₹ 1,00,00,000 - ₹ 1,20,00,000
Job Title: Senior Web Scraping Engineer

We are seeking a skilled professional to join our data team as a Senior Web Scraping Engineer. As a key member of the team, you will be responsible for designing and implementing a high-throughput product data ingestion pipeline across hundreds of domains.

Main Responsibilities:
  • Design an HTTP-first crawler with Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs for incremental runs.
  • Build a lightweight 'needs JS?' classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles/backoff and add URL normalization/canonicalization and de-duplication.
  • Handle PDF discovery & download with SHA-256 keys and apply Playwright browser automation resource budgets.
  • Integrate third-party APIs as first-class sources and own automation & orchestration for scheduled runs.
  • Create per-domain selectors with verification on hold-outs and maintain allow/deny paths.
Requirements:
  • 4+ years Python experience, including 2+ years building production web crawlers at scale.
  • Strong experience with Scrapy or aiohttp/asyncio and Playwright (or Puppeteer) in production.
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery for scheduled runs and monitoring.
  • Pdf handling and file integrity checks.
  • Queues (Redis/Kafka), Docker, Linux basics; comfort with logs/metrics.
  • Clear, pragmatic communication and strong ownership.
Nice to Have:
  • Go or Node.js experience for high-performance crawlers.
  • Cloud: AWS/GCP, S3, ECS/Kubernetes; IaC basics.
  • Workflow engines: Airflow/Temporal/Argo/Celery.
  • Document extraction: Textract/Tika/Camelot/Tabula.
  • Search/analytics: Elasticsearch/OpenSearch; warehousing (Snowflake/Postgres).
  • LLM-assisted selector generation with deterministic verification.

This role requires a strong understanding of web scraping, crawling, and API integration. You will work closely with our data team to design and implement a scalable data ingestion pipeline.

Key Skills: Web Scraping, Crawling, API Integration, Data Ingestion Pipeline, Scalability, Cloud Computing, Containerization, Linux, Python, JavaScript, CSS, XPath, JSON-LD, HTML Parsing, Proxy Management, Anti-Bot Tactics, Rate Limiting, File Integrity Checks, Log Analysis, Metrics Monitoring.

Bonus Skills: Go, Node.js, AWS, GCP, S3, ECS, Kubernetes, Airflow, Temporal, Argo, Celery, Textract, Tika, Camelot, Tabula, Elasticsearch, OpenSearch, Snowflake, Postgres, LLM-Assisted Selector Generation.

  • Senior Data Management Specialist

    2 weeks ago

    Nellore, Andhra Pradesh, India beBeeDataManagement Full time ₹ 15,00,000 - ₹ 25,00,000

    Senior Data Management SpecialistAbout the Role:We are seeking an experienced Senior Data Management Specialist to join our team. The ideal candidate will have hands-on experience in managing static/reference data and working with vendors such as Bloomberg, Refinitiv, and ICE.The key responsibilities of this role include capturing business requirements,...

  • Senior Production Support Specialist

    1 week ago

    Nellore, Andhra Pradesh, India beBeeProductionSupport Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job Title: Senior Production Support SpecialistOverviewThis is a senior-level position in production support, focusing on capital market experience.Key ResponsibilitiesTo provide direct business user support at an L2 levelTo work with SQL/Unix and have hands-on experience with VB/Powerquery/PythonTo manage monitoring and job scheduling tools like Control-MTo...

  • Quality Focused Production Specialist

    5 days ago

    Nellore, Andhra Pradesh, India beBeeProductivity Full time ₹ 80,00,000 - ₹ 1,00,00,000

    As a senior production quality specialist, you will be responsible for driving the implementation of quality and safety standards across our suppliers and manufacturing units to ensure optimal output of our products.Key Responsibilities:Promote the development and production of high-quality goods in line with our strategic objectives and global performance...

  • Senior Data Integration Specialist

    2 weeks ago

    Nellore, Andhra Pradesh, India beBeeDataIntegration Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Senior Data Integration SpecialistWe are seeking an experienced professional with a strong background in cloud data integration, application integration, and master data management to join our team.Develop and configure high-performance integrations using file-based, FTP/SFTP, APIs, real-time, and batch processes.Implement and troubleshoot complex...

  • Senior FICO Specialist

    2 weeks ago

    Nellore, Andhra Pradesh, India beBeeFicoSpecialist Full time ₹ 1,80,00,000 - ₹ 3,00,00,000

    Job Title: Senior FICO SpecialistWe are seeking an experienced Senior FICO specialist to join our team. The successful candidate will have a strong background in financial modeling and analysis, with experience working with manufacturing companies.Minimum 10 years of relevant experience in financial modeling and analysisExperience working with manufacturing...

  • Senior Billing System Specialist

    2 days ago

    Nellore, Andhra Pradesh, India beBeeBilling Full time ₹ 1

    Senior Billing System SpecialistThis is an exceptional opportunity to showcase your expertise in billing system development.We are seeking a highly skilled Senior Billing System Specialist to join our team on a contract basis. The selected candidate will be responsible for providing production support, troubleshooting issues, and implementing billing...

  • Senior Investment Fund Specialist

    6 days ago

    Nellore, Andhra Pradesh, India beBeeInvestment Full time ₹ 90,00,000 - ₹ 1,20,00,000

    Senior Investment Fund Specialist Job Overview We are seeking a highly skilled professional with expertise in alternative investment funds to fill this key role. As a Senior Investment Fund Specialist, you will serve as a subject matter expert for analyzing fund-level agreements between general partners and limited partners. You will oversee the accuracy and...

  • Senior Data Infrastructure Specialist

    1 week ago

    Nellore, Andhra Pradesh, India beBeeDataInfrastructure Full time ₹ 15,00,000 - ₹ 25,00,000

    Unlock the potential of our database systems by joining a dynamic team as a Senior Data Infrastructure SpecialistThis key role brings together traditional database administration skills with modern data engineering tasks to drive the performance and scalability of our critical database systems.Data Architecture & ScalabilityRedesign and improve JSON blob...

  • Senior Data Infrastructure Specialist

    6 days ago

    Nellore, Andhra Pradesh, India beBeeInfrastructure Full time ₹ 18,49,999 - ₹ 20,99,999

    Job Description:We are seeking an experienced Senior Data Infrastructure Specialist to join our team. The successful candidate will be responsible for designing, building, and operating scalable data infrastructure to support large-scale machine learning research.The ideal candidate will have a strong background in distributed computing frameworks, cloud...

  • Senior Advertising Specialist

    2 weeks ago

    Nellore, Andhra Pradesh, India beBeeAdvertising Full time ₹ 6,00,000 - ₹ 12,00,000

    Job Title: Senior Advertising SpecialistJob Description:We are seeking an experienced and results-driven Senior Advertising Specialist to manage and optimize advertising campaigns, ensuring maximum return on investment (ROI) for our e-commerce business.Key Responsibilities:Develop, execute, and optimize Meta Ads (Facebook & Instagram) for e-commerce brands,...