Senior Backend Engineer – High-Performance Web Crawling

4 weeks ago

Jamnagar, Gujarat, India Magma Consultancy Full time

Location: [Remote / India]

Job Type: [Full-time]

Experience: 5+ years in web crawling/scraping, backend systems, and data extraction

About Client

Our Client is a modern, meaning‑based web-search API designed specifically for AI applications—such as retrieval‑augmented generation (RAG). Unlike traditional keyword-based engines, they use embedding-based semantic search, allowing developers to fetch content that's contextually relevant and up-to-date.

About the Opportunity

Our client is building an ambitious new platform designed to push the boundaries of web-scale search and AI. They are developing a custom in-house crawler and vector database that will serve as the foundation for next-generation semantic search and retrieval.

To support this, they are hiring a Senior Backend Engineer to lead efforts on a distributed crawler capable of handling over 3 million URLs per hour. This role is ideal for someone who thrives on scale, performance, and elegant engineering solutions.

Key Responsibilities

  • Architect and scale a distributed, asynchronous web crawler capable of processing millions of URLs per hour
  • Design and build systems for fetching, parsing, deduplicating, and storing web content efficiently
  • Build robust pipelines that integrate raw crawl data into an in-house vector search index.
  • Optimize across the stack: network I/O, concurrency, storage, and CPU utilization.
  • Ensure resilience and fault tolerance in a globally distributed system
  • Work closely with infrastructure and ML teams to embed, classify, and store crawled content in real-time
  • Monitor performance and system health using modern observability tooling

Preferred Technical Experience

  • 5+ years of backend/systems engineering experience
  • Strong understanding of concurrent programming, async I/O, and distributed architecture
  • Experience with web crawling frameworks (e.g., Colly, Scrapy, aiohttp) or building crawlers from scratch
  • Deep familiarity with HTTP, DNS, TLS, rate limiting, and error handling at scale
  • Proficiency in Go, Rust, or Python (async); willingness to learn as needed
  • Comfort working with message queues, distributed storage, and container orchestration tools (e.g., Kafka, Redis, Kubernetes)
  • Exposure to vector databases (Weaviate, Qdrant, FAISS, etc.) is a strong plus
  • Knowledge of anti-bot mechanisms, proxy strategies, and adaptive crawling techniques

What Makes This Unique

  • Chance to work on planet-scale infrastructure from day one
  • An ambitious, engineering-first culture that values speed, precision, and experimentation
  • No red tape, no legacy systems — only greenfield challenges and impact
  • Work directly with a founding team that has built and scaled products used by millions
  • Competitive compensation, equity, and a remote-first work environment

Location

Remote (global) — team is distributed across multiple time zones.

Interested? Apply with updated CV & Portfolio on rajabhattacharya@magmaconsultancy.in

  • High-Performance Backend Engineer

    2 days ago

    Jamnagar, Gujarat, India beBeeBackend Full time US$ 1,40,000 - US$ 1,90,000

    Job DescriptionWe are seeking a seasoned backend engineer to collaborate with our engineering team in designing, building and maintaining scalable high-performance backend systems.The ideal candidate will contribute to the architecture, development, and operations of mission-critical services that support our platform.Key ResponsibilitiesDesign and build...

  • High-Performance Backend Specialist

    6 hours ago

    Jamnagar, Gujarat, India beBeeDeveloper Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Backend Developer RoleAbout the PositionWe are looking for an experienced Backend Developer to create scalable backend services using Spring Boot and microservices.Key Responsibilities:Design, develop, and optimize backend services using Java, Spring Boot, and Microservices.Build and maintain RESTful APIs for high performance, security, and scalability.Work...

  • High-Performance Data Engineer

    3 days ago

    Jamnagar, Gujarat, India beBeeData Full time ₹ 60,00,000 - ₹ 80,00,000

    We are seeking a talented data professional to lead our efforts in designing, building and maintaining high-performance data pipelines.Role OverviewThis role is critical in ensuring our data infrastructure can handle increasing volumes of data while maintaining exceptional performance standards.Key Responsibilities:Design and implement scalable batch...

  • Senior Backend Web Application Developer

    11 hours ago

    Jamnagar, Gujarat, India beBeeDeveloper Full time ₹ 20,00,000 - ₹ 25,00,000

    Full Stack Java DeveloperWe are seeking a highly skilled Full Stack Java Developer to join our dynamic team. The ideal candidate will have over 6 years of hands-on experience in designing, developing, and maintaining scalable web applications using Java on the backend and React with Redux and TypeScript on the frontend.Key Responsibilities:Design and develop...

  • AI Systems Developer

    1 day ago

    Jamnagar, Gujarat, India beBeeAutomation Full time ₹ 50,000 - ₹ 70,000

    **Job Title:** Senior AI Developer Job Description:We are seeking a senior experienced AI/ML developer to design and build an intelligent system that can interact with websites, perform smart actions (clicking, downloading, form filling, navigation), and extract data automatically. The role involves combining AI techniques with web crawling/automation tools...

  • Senior Backend Engineer

    3 days ago

    Jamnagar, Gujarat, India beBeeEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    About LeapWe are building a new 0-1 stablecoin payments initiative.Stablecoins are one of the fastest growing categories in crypto, with clear product market fit in payments and cross-border use cases.The role is focused on leading engineering for the entire stablecoin initiative – spanning both application and infrastructure layers.You will lead backend...

  • Senior Software Backend Engineer

    6 hours ago

    Jamnagar, Gujarat, India beBeeBackendDeveloper Full time ₹ 20,00,000 - ₹ 30,00,000

    At the forefront of AgTech innovation, we're driving the future of farming through cutting-edge technology and expert collaboration.Key Responsibilities:Design, develop, and maintain robust backend systems and microservices that meet the highest standards of performance and reliability.Develop and optimize RESTful and GraphQL APIs to deliver high-performance...

  • Senior Backend Developer Opportunity

    6 hours ago

    Jamnagar, Gujarat, India beBeeBackend Full time ₹ 15,00,000 - ₹ 25,00,000

    About UsWe are a forward-thinking organization founded by seasoned entrepreneurs with a proven track record of success. Our mission is to make high-quality education accessible and affordable for students worldwide, empowering the next generation of leaders.Role OverviewAs a Backend Engineer, you will be part of our core engineering team, working closely...

  • Highly Skilled Golang Backend Developer Wanted

    17 hours ago

    Jamnagar, Gujarat, India beBeeBackend Full time ₹ 15,00,000 - ₹ 20,00,000

    Senior Golang Backend Engineer PositionWe are seeking a highly skilled and experienced Golang backend engineer to join our engineering team. The ideal candidate will have expertise in building reliable, scalable systems and writing clean, maintainable code.We require:Proficiency in backend programming languages such as Node.js, Python, or GolangExperience...

  • High Performance Software Architect

    1 day ago

    Jamnagar, Gujarat, India beBeeBackend Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    **Job Title: Lead Backend Developer**Ambitious and experienced software engineers with a passion for scaling up complex architectures are encouraged to apply for this challenging role. We are seeking a highly skilled Lead Backend Developer to join our dynamic engineering team.The successful candidate will be responsible for architecting the...