Lead Data Extractor

12 hours ago

Mysore, Karnataka, India beBeeWebScraping Full time ₹ 20,00,000 - ₹ 25,00,000
Senior Web Scraping Engineer Job

We are seeking a highly skilled Senior Web Scraping Engineer to join our team.

About the Role:

Our organization is developing a high-throughput product data ingestion pipeline across hundreds of domains. As a Senior Web Scraping Engineer, you will be responsible for designing and implementing an HTTP-first crawler with a Playwright fallback for JS-heavy pages.

Key Responsibilities:
  • Design an HTTP-first crawler using Scrapy or aiohttp with Playwright fallback for JS-heavy pages.
  • Implement sitemap diffing and conditional GETs (ETag/Last-Modified) for incremental runs.
  • Build a lightweight classifier to auto-route HTTP vs Playwright.
  • Enforce per-domain throttles/backoff and add URL normalization/canonicalization.
  • Handle PDF discovery & download, apply Playwright browser automation resource budgets, and integrate third-party APIs as first-class sources.
  • Own automation & orchestration for scheduled runs, idempotent retries, and alerting.
  • Ship observability metrics, maintain allow/deny paths, and adhere to robots.txt and Terms of Service.
Mandatory Qualifications:
  • 4+ years Python experience, including 2+ years building production web crawlers at scale.
  • Strong expertise in Scrapy or aiohttp/asyncio and Playwright (or Puppeteer) in production.
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL (auth, pagination, backoff) and building small internal services.
  • Automation/Orchestration: Airflow/Temporal/Celery (or equivalent schedulers/queues) for scheduled runs and monitoring.
  • PDF handling (requests/HEAD, hashing, size limits) and file integrity checks.
  • Queues (Redis/Kafka), Docker, Linux basics; comfort with logs/metrics.
  • Clear, pragmatic communication and strong ownership.

  • Expert Data Solutions Architect

    2 weeks ago

    Mysore, Karnataka, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job DescriptionAs a Technical Consultant, you will play a pivotal role in configuring and optimizing financial data workflows within the SimCorp Dimension platform.The role demands hands-on expertise in SimCorp tools, strong XSLT and XML development skills, and a solid understanding of asset and portfolio management systems.Lead configuration efforts across...

  • Data Transformation Specialist

    4 days ago

    Mysore, Karnataka, India beBeeBackend Full time ₹ 15,00,000 - ₹ 25,00,000

    Transform Data into Insights">We are seeking a skilled Backend Engineer with experience in Go or Rust to contribute to our core infrastructure. You will play a key role in decoding smart contract events and implementing pricing logic from decentralized exchanges.">This role is ideal for someone who enjoys low-level protocol work, understands DeFi mechanics,...

  • Advanced Data Engineer

    2 days ago

    Mysore, Karnataka, India beBeeBackend Full time US$ 80,000 - US$ 1,60,000

    Job Overview:We are seeking a skilled Backend Developer with experience in Go or Rust to join our team. The ideal candidate will play a key role in transforming raw on-chain data into actionable insights by decoding smart contract events and implementing pricing logic from decentralized exchanges (DEXs).This role is suitable for individuals who enjoy...

  • Data Engineering Lead

    2 days ago

    Mysore, Karnataka, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Data Engineering LeadJob Description:We are seeking an experienced Data Engineering Lead to join our team. The successful candidate will be responsible for designing, developing, and implementing data engineering solutions that meet the needs of our business.Key Responsibilities:Lead the development of data engineering architectures and...

  • Lead Data Engineer

    1 day ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 2,50,00,000 - ₹ 3,20,00,000

    Engineering Manager - Data PositionThis is an exciting opportunity to lead cross-functional teams in delivering complex data engineering initiatives using agile methods and frameworks.About the RoleWe are seeking a skilled Engineering Manager to join our organization. The successful candidate will be responsible for providing technical leadership, guidance,...

  • Data Science Lead

    4 days ago

    Mysore, Karnataka, India beBeeMachineLearning Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    We are seeking a highly skilled Data Science Lead to drive business growth through data-driven insights.About the Role:The ideal candidate will have a strong background in machine learning, deep learning, and artificial intelligence techniques. They will be responsible for developing and implementing advanced analytics methods to extract value from business...

  • Data Governance Lead

    7 days ago

    Mysore, Karnataka, India beBeeDataGovernance Full time US$ 1,80,000 - US$ 2,50,000

    Job OverviewThe lead of Data Governance will oversee the development and implementation of strategies for managing data across marketing and consumer engagement initiatives.

  • Data Analytics Lead

    2 days ago

    Mysore, Karnataka, India beBeeDataAnalytics Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Analytics LeadAs a key member of our team, you will lead the development and implementation of predictive models that drive business growth and customer satisfaction.The ideal candidate will have expertise in machine learning, statistical modeling, and data analysis. They will be able to communicate complex ideas effectively to both technical...

  • Senior Data Engineering Lead

    1 week ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,10,000

    Job OverviewThis is a senior data engineering position that involves leading a team of big data engineers.The ideal candidate will have excellent problem analysis skills and experience with at least one cloud infra provider such as Azure or AWS.They will also be proficient in building data pipelines using batch processing with Apache Spark or Hive query...

  • Data Warehouse Development Lead

    1 week ago

    Mysore, Karnataka, India beBeeDevelopment Full time ₹ 15,00,000 - ₹ 25,00,000

    Lead Data Warehouse Development TeamWe seek an experienced professional to oversee the design, development, and configuration of data warehouse applications.The ideal candidate will have strong expertise in data warehousing with at least 5 years of hands-on experience. They will be responsible for collaborating with cross-functional teams, overseeing project...