Senior Backend Engineer – High-performance Web Crawling

4 weeks ago

India Magma Consultancy Full time
Location: (Remote / India)Job Type: (Full-time)Experience: 5+ years in web crawling/scraping, backend systems, and data extraction About Client Our Client is a modern, meaning‑based web-search API designed specifically for AI applications—such as retrieval‑augmented generation (RAG).

Unlike traditional keyword-based engines, they use embedding-based semantic search, allowing developers to fetch content that's contextually relevant and up-to-date.

About the Opportunity Our client is building an ambitious new platform designed to push the boundaries of web-scale search and AI.

They are developing a custom in-house crawler and vector database that will serve as the foundation for next-generation semantic search and retrieval.

To support this, they are hiring a Senior Backend Engineer to lead efforts on a distributed crawler capable of handling over 3 million URLs per hour.

This role is ideal for someone who thrives on scale, performance, and elegant engineering solutions.

Key Responsibilities Architect and scale a distributed, asynchronous web crawler capable of processing millions of URLs per hour Design and build systems for fetching, parsing, deduplicating, and storing web content efficiently Build robust pipelines that integrate raw crawl data into an in-house vector search index.

Optimize across the stack: network I/O, concurrency, storage, and CPU utilization.

Ensure resilience and fault tolerance in a globally distributed system Work closely with infrastructure and ML teams to embed, classify, and store crawled content in real-time Monitor performance and system health using modern observability tooling Preferred Technical Experience5+ years of backend/systems engineering experience Strong understanding of concurrent programming, async I/O, and distributed architecture Experience with web crawling frameworks (e.g., Colly, Scrapy, aiohttp) or building crawlers from scratch Deep familiarity with HTTP, DNS, TLS, rate limiting, and error handling at scale Proficiency in Go, Rust, or Python (async); willingness to learn as needed Comfort working with message queues, distributed storage, and container orchestration tools (e.g., Kafka, Redis, Kubernetes)Exposure to vector databases (Weaviate, Qdrant, FAISS, etc.) is a strong plus Knowledge of anti-bot mechanisms, proxy strategies, and adaptive crawling techniques What Makes This Unique Chance to work on planet-scale infrastructure from day one An ambitious, engineering-first culture that values speed, precision, and experimentation No red tape, no legacy systems — only greenfield challenges and impact Work directly with a founding team that has built and scaled products used by millions Competitive compensation, equity, and a remote-first work environment Location Remote (global) — team is distributed across multiple time zones.

Interested? Apply with updated CV & Portfolio on rajabhattacharya@magmaconsultancy.in

  • Senior Backend Engineer

    4 weeks ago

    India Magma Consultancy Full time

    Location: [Remote / India] Job Type: [Full-time] Experience: 5+ years in web crawling/scraping, backend systems, and data extraction About Client Our Client is a modern, meaning‑based web-search API designed specifically for AI applications—such as retrieval‑augmented generation (RAG).Unlike traditional keyword-based engines, they use embedding-based...

  • High-Performance Backend Engineer

    11 hours ago

    India beBeeBackend Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Backend Engineer RoleAt our organization, we're seeking a skilled Backend Engineer to contribute to building and scaling high-performance backend systems.Key Responsibilities:Design, develop, and maintain scalable backend services and APIs using Golang.Collaborate with cross-functional teams to deliver high-impact features and enhance the overall user...

  • High-Performance Software Engineer

    18 hours ago

    India beBeeBackendDeveloper Full time ₹ 9,00,000 - ₹ 12,00,000

    Senior Backend DeveloperWe are seeking a highly skilled Senior Backend Developer to join our team. This is an exciting opportunity to work on building scalable, high-performance applications using Python (Django), Node.js, and React.**Key Responsibilities:**- Architect, develop, and maintain backend services using Python (Django) & Node.js- Design and...

  • High-Performance Web Developer

    6 days ago

    India beBeeDevelopment Full time ₹ 6,00,000 - ₹ 10,00,000

    As a high-performance web developer, you will be responsible for crafting tailored solutions to drive business growth.Key ResponsibilitiesDesign and develop responsive Shopify themes that perfectly balance backend architecture with a polished frontend experience.Coding, debugging, testing, and refining frontend and backend functionalities are daily...

  • Senior Backend Engineer

    2 days ago

    India Xebia Full time

    Senior Backend Engineer (Python Fast API, Dev Ops) | Xebia???? Locations: Chennai, Bangalore, Hyderabad, Pune, Bhopal, Gurugram, Jaipur (Hybrid – 3 days/week office)???? Experience: 10+ Years???? Immediate joiners to max 2 weeks' notice onlyAbout the RoleWe are looking for a Senior Backend Engineer with strong Dev Ops expertise to join our team.The ideal...

  • Backend Engineer

    4 weeks ago

    India Zendr Business Full time

    Backend Developer (Node.js & Express.js) - Full-Time (Remote, India)Company: Zendr ltd, UK.Location: Remote (India-based candidates only)Experience Required: Minimum 3 years Salary: To be discussed About Zendr: Zendr is a UK-based startup on a mission to transform how businesses manage payments, invoicing, and operations.We're building the future of business...

  • Senior Backend Engineer

    3 weeks ago

    India Infinite Possibilities Full time

    About UsAt GenAI Labs IP India Pvt Ltd, we're reimagining how Private Equity and consulting teams work—by building intelligent AI-native workflows and agentic platforms tailored to the real-world needs of high-performance teams. Our mission is to accelerate value creation through cutting-edge AI, system design, and product craftsmanship. We're a...

  • High-Performance Software Engineer

    2 days ago

    India beBeeBackend Full time US$ 80,000 - US$ 1,20,000

    Expertise AmplifiedWe are building a cutting-edge platform that leverages artificial domain intelligence to transform expertise. Our mission is to empower experts to harness AI without complexity, allowing them to focus on their core strengths.Role Overview:As a Backend Engineer, you will play a pivotal role in designing, developing, and maintaining...

  • Senior Backend Engineer

    3 weeks ago

    India Infinite Possibilities Full time

    About UsAt GenAI Labs IP India Pvt Ltd, we're reimagining how Private Equity and consulting teams work—by building intelligent AI-native workflows and agentic platforms tailored to the real-world needs of high-performance teams. Our mission is to accelerate value creation through cutting-edge AI, system design, and product craftsmanship. We're a...

  • Senior Backend Engineer

    3 days ago

    India beBeeBackend Full time ₹ 8,00,000 - ₹ 12,00,000

    Job Title: Senior Backend Engineer\We are seeking a highly skilled and experienced Senior Backend Engineer to join our team. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining scalable and efficient backend systems.\Key Responsibilities:\\Design and develop high-quality backend systems using Java,...