High Performance Data Engineer

2 weeks ago

Agra, Uttar Pradesh, India beBeeEngineering Full time ₹ 10,00,000 - ₹ 15,00,000
Senior Data Engineering Role

About the Position:

We are seeking a highly skilled Senior Data Engineer to lead our data ingestion pipeline. This individual will be responsible for designing and implementing an efficient data crawling and extraction system, utilizing HTTP-first crawling with a Playwright fallback, per-domain learned selectors, and reliable PDF handling.

This role involves both crawling (discovering and fetching pages via sitemaps/robots) and scraping (extracting structured specs, images, and PDFs into our schema). Key responsibilities include designing an HTTP-first crawler with Playwright fallback, implementing sitemap diffing and conditional GETs, building a lightweight classifier to auto-route HTTP vs Playwright, and enforcing per-domain throttles/backoff.

The successful candidate will also integrate third-party APIs as first-class sources, handle authentication, pagination, and rate limits, and unify API + crawl outputs. Additionally, they will own automation & orchestration for scheduled runs, idempotent retries, and alerting, create per-domain selectors with verification on hold-outs, and ship observability metrics.

Maintenance of allow/deny paths, adherence to robots.txt and Terms of Service, containerization of workers, provision of runbooks/CI, and collaboration with data team on schemas/normalization are crucial aspects of this position.

Requirements:
  • 4+ years Python experience, including 2+ years building production web crawlers at scale.
  • Strong experience with Scrapy or aiohttp/asyncio and Playwright (or Puppeteer) in production.
  • Practical proxy management, polite anti-bot tactics, and per-domain rate limiting.
  • Hands-on with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL (auth, pagination, backoff) and building small internal services (FastAPI or similar).
  • Automation/Orchestration: Airflow/Temporal/Celery (or equivalent schedulers/queues) for scheduled runs and monitoring.
  • PDF handling (requests/HEAD, hashing, size limits) and file integrity checks.
  • Queues (Redis/Kafka), Docker, Linux basics; comfort with logs/metrics.
  • Clear, pragmatic communication and strong ownership.
Benefits:
  • Competitive compensation package.
  • Ongoing professional development opportunities.
  • A dynamic, collaborative work environment.
How We Work:
  • We prioritize simple designs that are easy to operate at scale.
  • Track coverage and freshness as key performance indicators.
  • We value clear, concise communication and strong teamwork.

  • High-Performance Data Architect Leader

    2 weeks ago

    Agra, Uttar Pradesh, India beBeeData Full time ₹ 1,44,00,000 - ₹ 2,02,50,000

    Data Architect LeadAre you an experienced data professional seeking a new challenge? We are looking for a skilled Data Architect Lead to design and implement scalable, high-performance data pipelines using Snowflake and dbt.This role is ideal for someone with a strong background in data architecture, who can define architectural best practices and drive data...

  • High-Performance Search Engineer

    1 week ago

    Agra, Uttar Pradesh, India beBeeEngineer Full time ₹ 1,50,00,000 - ₹ 2,25,00,000

    We are seeking an experienced Information Retrieval Specialist to design, develop, and optimize search solutions that deliver exceptional user experiences.The ideal candidate will combine strong software engineering skills with deep knowledge of information retrieval systems. They will be responsible for designing and implementing advanced search...

  • High-Performing Site Reliability Engineer Wanted

    2 weeks ago

    Agra, Uttar Pradesh, India beBeeEngineer Full time ₹ 25,00,000 - ₹ 31,25,000

    Site Reliability Engineer OpportunityWe are seeking a highly skilled Site Reliability Engineer to ensure the stability, scalability, and operational excellence of our Accounting and Finance platforms.Key Responsibilities:Ensure Accounting and Finance platforms meet defined SLAs, SLOs, and SLIs for performance, reliability, and uptime.Build automation for...

  • High Performing PPC Specialist

    2 weeks ago

    Agra, Uttar Pradesh, India beBeePPC Full time  50,00,000 -  70,00,000

    High-Performing PPC Specialist SoughtWe're looking for a dedicated professional to oversee our paid search campaigns and drive significant revenue growth.This role requires a deep understanding of search engine advertising platforms, as well as experience in leading, mentoring, and developing high-performing teams.The ideal candidate will have a proven...

  • High-Performance Data Professional

    2 weeks ago

    Agra, Uttar Pradesh, India beBeeData Full time ₹ 12,00,000 - ₹ 25,00,000

    Job Title: Data DeveloperWe are seeking a highly skilled and versatile data developer to execute various tasks within our organization.Key Responsibilities:Write efficient, optimized SQL queries to extract, transform, and aggregate data for analytical purposes.Develop in-depth understanding of ETL/ELT processes and data flow from source systems into...

  • High-Performing Automation Expert

    2 weeks ago

    Agra, Uttar Pradesh, India beBeeEngineer Full time ₹ 30,00,000 - ₹ 45,00,000

    Job Title: Performance Test EngineerIndustry:IT Services and IT Consulting Job DescriptionThe ideal candidate will be responsible for developing high-quality applications.About Company :We guide customers from what's now to what's next by unlocking the value of their data and applications to solve their digital challenges, achieving outcomes that benefit...

  • High-Performance Processor Core Engineer

    2 weeks ago

    Agra, Uttar Pradesh, India beBeeLogic Full time ₹ 1,50,00,000 - ₹ 2,25,00,000

    Senior Logic Design EngineerWe are seeking a skilled Senior Logic Design Engineer to lead the development of high-performance processor core front-end pipeline units.The ideal candidate will have experience architecting and designing specific CPU units, such as I-Cache, Instruction Fetch, Branch Prediction, and Instruction Decode.This role requires a deep...

  • High-Performing Systems Engineer

    2 weeks ago

    Agra, Uttar Pradesh, India beBeePerformance Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Transforming PerformanceAbout UsWe are a global digital solutions company driven by innovation and value for clients across industries. Our mission is to drive meaningful change through the power of technology.Job Overview:The Performance Engineering Specialist is responsible for ensuring that software systems perform optimally under various load conditions....

  • High Performance Engineer for Mid-Tier Services and Databases

    2 weeks ago

    Agra, Uttar Pradesh, India beBeePerformanceEngineer Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    **Job Description:**We are seeking a high performance engineer to work with our development teams. The successful candidate will be responsible for creating performance test plans, test scenarios, test scripts, test execution, results analysis, and providing insight into potential performance issues.The ideal candidate will have mandatory experience with...

  • High-Performance Software Developer

    1 week ago

    Agra, Uttar Pradesh, India beBeeDevelopment Full time US$ 1,20,000 - US$ 1,80,000

    Our company specializes in leveraging innovation and technology to connect businesses with top software talent.Our growth is driven by the increasing demand for high-quality professionals from leading companies.Job Title: Frontend DeveloperExperience: 5+ yrsJob Description:Key Responsibilities:Design and develop complex algorithms using JavaScript...