Experienced Web Scraping Specialist

2 weeks ago

Varanasi, Uttar Pradesh, India beBeeWebScraping Full time ₹ 19,00,800 - ₹ 24,14,000
Senior Web Scraping Engineer

Overview:

We are seeking an experienced Senior Web Scraping Engineer to join our team. As a key member of our engineering department, you will be responsible for building and maintaining high-performance web crawling systems that extract structured data from websites.

The ideal candidate will have strong technical skills in Python, Scrapy, aiohttp, and Playwright. You should also have experience with cloud platforms such as AWS or GCP.

Key Responsibilities:
  • Design and implement high-throughput web crawlers using Scrapy or aiohttp.
  • Develop and maintain per-domain selectors using YAML and HTML parsing techniques.
  • Implement sitemap diffing and conditional GETs for incremental runs.
  • Handle rate limiting and retries using ETag/Last-Modified and backoff algorithms.
  • Integrate third-party APIs using REST and GraphQL protocols.

Additionally, the successful candidate will have experience with automation and orchestration tools such as Airflow, Temporal, and Celery. They should also be comfortable working with PDF handling, file integrity checks, and queueing systems like Redis and Kafka.

Requirements:
  • 4+ years of experience in software development with a focus on web scraping.
  • Strong understanding of Python, Scrapy, aiohttp, and Playwright.
  • Experience with cloud platforms such as AWS or GCP.
  • Knowledge of API protocols such as REST and GraphQL.
  • Hands-on experience with automation and orchestration tools.

As a Senior Web Scraping Engineer, you will have the opportunity to work on challenging projects and collaborate with a talented team of engineers. We offer competitive compensation and benefits packages, including opportunities for professional growth and development.

How to Apply:

To apply for this exciting opportunity, please submit your resume and cover letter along with links to relevant repositories or code samples. In your application, please provide concise notes on:

  1. A crawler you have built and run at scale (100+ sites/day).
  2. How you handled rate limits and retries.
  3. Your approach to PDF discovery and deduplication.

  • Web Interface Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeefrontend Full time ₹ 35,00,000 - ₹ 75,00,000

    Front-End Web Developer PositionAbout the Role:We are seeking a highly skilled Front-End Web Developer to join our team. The successful candidate will be responsible for designing and developing responsive and interactive user interfaces for our web applications.Key Responsibilities:Develop web interfaces using HTML, CSS, and JavaScriptCreate UI components...

  • Innovative Data Solutions Developer

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 15,00,000

    We are seeking an entrepreneurial, passionate and driven individual to shape our technology vision, architecture and engineering culture from the ground up.This role is ideal for someone eager to own the entire tech stack, who thrives on early-stage challenges, and loves building innovative, scalable solutions from day zero.Key ResponsibilitiesData...

  • Web User Acquisition Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeAffiliate Full time ₹ 40,00,000 - ₹ 80,00,000

    About the RoleWe seek a results-driven affiliate manager with a strong focus on web-based user acquisition.The ideal candidate will be responsible for managing, growing, and optimizing our affiliate and publisher network to drive quality leads, conversions, and revenue for our web campaigns (CPL/CPA/CPS models).

  • Senior Web Interface Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeFrontend Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Title: UI/TL DeveloperExperience: 3 to 5 YearsJob Overview:We are seeking a creative and skilled Front-end Developer with 3+ years of hands-on experience in crafting responsive, user-friendly web interfaces. Key strengths include modern front-end technologies such as HTML5, CSS3, JavaScript (ES6+), React.js, and UI frameworks like Material-UI.Key...

  • Leading Software QA

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeQuality Full time ₹ 18,00,000 - ₹ 24,00,000

    Lead Quality Assurance SpecialistWe are seeking an experienced Lead Quality Assurance Specialist to oversee the quality assurance efforts for our web domain project.The successful candidate will be responsible for developing test strategies, plans, and processes for web applications, leading and managing the QA team, including mentoring and providing...

  • Digital Marketing Specialist

    3 days ago

    Varanasi, Uttar Pradesh, India Fish White Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Company DescriptionFish White, founded in 2022 and based in Varanasi, is a startup in the fish industry dedicated to delivering live fresh white meat to consumers. We focus on ensuring that all our products reach the end consumer in the freshest state possible. As a brand, Fish White aims to provide a hassle-free, healthy meat experience. Our mission is to...

  • Digital Marketing Specialist

    3 days ago

    Varanasi, Uttar Pradesh, India Honelogix web solutions Full time ₹ 1,20,000 - ₹ 1,80,000 per year

    Position: Digital Marketing ExecutiveDepartment: MarketingLocation: VaranasiJob Type: Full-TimeAbout the RoleWe are looking for a passionate and motivated Digital Marketing Specialist to join our growing team. The candidate will assist in planning, implementing, and monitoring our digital marketing campaigns across various channels including SEO, SMO, PPC,...

  • Creative Content Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeContentCreator Full time ₹ 15,00,000 - ₹ 20,00,000

    About the RoleAs a content creation specialist, you will be responsible for crafting compelling marketing campaigns that resonate with our target audience. To excel in this position, you must possess excellent writing skills and a deep understanding of the English language.Our Ideal CandidateWe are seeking a highly collaborative individual who can...

  • Experienced Security Professionals Wanted

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeesecurity Full time ₹ 1,85,00,000 - ₹ 2,53,50,000

    About the RoleWe are seeking a highly skilled and experienced professional to fill the position of Staff Security Engineer. The ideal candidate will possess expertise in application security, penetration testing, and offensive security practices.This is a hands-on technical role with significant influence over the company's security posture, from code to...

  • Senior Front End Specialist

    2 weeks ago

    Varanasi, Uttar Pradesh, India beBeeDeveloper Full time ₹ 30,00,000 - ₹ 50,00,000

    Full Stack Web Developer OpportunityUnlock your full potential as a Full Stack Web Developer by contributing to the design and development of web applications.We are seeking an ambitious and skilled individual to join our team and work on both front-end and back-end tasks depending on their skillset, collaborating with our team to implement features like...