Senior Backend Engineer

4 weeks ago

India Magma Consultancy Full time
Location: [Remote / India] Job Type: [Full-time] Experience: 5+ years in web crawling/scraping, backend systems, and data extraction About Client Our Client is a modern, meaning‑based web-search API designed specifically for AI applications—such as retrieval‑augmented generation (RAG).
Unlike traditional keyword-based engines, they use embedding-based semantic search, allowing developers to fetch content that's contextually relevant and up-to-date.
About the Opportunity Our client is building an ambitious new platform designed to push the boundaries of web-scale search and AI.
They are developing a custom in-house crawler and vector database that will serve as the foundation for next-generation semantic search and retrieval.
To support this, they are hiring a Senior Backend Engineer to lead efforts on a distributed crawler capable of handling over 3 million URLs per hour.
This role is ideal for someone who thrives on scale, performance, and elegant engineering solutions.
Key Responsibilities Architect and scale a distributed, asynchronous web crawler capable of processing millions of URLs per hour Design and build systems for fetching, parsing, deduplicating, and storing web content efficiently Build robust pipelines that integrate raw crawl data into an in-house vector search index.
Optimize across the stack: network I/O, concurrency, storage, and CPU utilization.
Ensure resilience and fault tolerance in a globally distributed system Work closely with infrastructure and ML teams to embed, classify, and store crawled content in real-time Monitor performance and system health using modern observability tooling Preferred Technical Experience 5+ years of backend/systems engineering experience Strong understanding of concurrent programming, async I/O, and distributed architecture Experience with web crawling frameworks (e.g., Colly, Scrapy, aio or building crawlers from scratch Deep familiarity with DNS, TLS, rate limiting, and error handling at scale Proficiency in Go, Rust, or Python (async); willingness to learn as needed Comfort working with message queues, distributed storage, and container orchestration tools (e.g., Kafka, Redis, Kubernetes) Exposure to vector databases ( Weaviate, Qdrant, FAISS, etc.
) is a strong plus Knowledge of anti-bot mechanisms, proxy strategies, and adaptive crawling techniques What Makes This Unique Chance to work on planet-scale infrastructure from day one An ambitious, engineering-first culture that values speed, precision, and experimentation No red tape, no legacy systems — only greenfield challenges and impact Work directly with a founding team that has built and scaled products used by millions Competitive compensation, equity, and a remote-first work environment Location Remote (global) — team is distributed across multiple time zones.
Interested? Apply with updated CV & Portfolio on rajabhattacharya@magmaconsultancy.in

  • Senior Backend Engineer

    3 hours ago

    India Xebia Full time

    Senior Backend Engineer (Python Fast API, Dev Ops) | Xebia???? Locations: Chennai, Bangalore, Hyderabad, Pune, Bhopal, Gurugram, Jaipur (Hybrid – 3 days/week office)???? Experience: 10+ Years???? Immediate joiners to max 2 weeks' notice onlyAbout the RoleWe are looking for a Senior Backend Engineer with strong Dev Ops expertise to join our team.The ideal...

  • Senior Backend Engineer

    1 day ago

    India beBeeBackend Full time ₹ 8,00,000 - ₹ 12,00,000

    Job Title: Senior Backend Engineer\We are seeking a highly skilled and experienced Senior Backend Engineer to join our team. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining scalable and efficient backend systems.\Key Responsibilities:\\Design and develop high-quality backend systems using Java,...

  • Senior Backend Engineer

    14 hours ago

    India nineDots Full time

    Senior Backend Engineer (Contract) – Initial 3 Months, Immediate Start Help shape the backend of a live, regulated financial platform being built across the Middle East. You'll join a lean team where the frontend is further along than the backend and the goal is to catch up. This is a full-time contract role (initial 3 months, possible extension) ...

  • Senior Backend Engineer

    1 day ago

    India nineDots Full time

    Senior Backend Engineer (Contract)– Initial 3 Months, Immediate StartHelp shape the backend of a live, regulated financial platform being built across the Middle East. You'll join a lean team where the frontend is further along than the backend and the goal is to catch up. This is a full-time contract role (initial 3 months, possible extension) focused on...

  • Senior Backend Engineer

    3 weeks ago

    India Infinite Possibilities Full time

    About UsAt GenAI Labs IP India Pvt Ltd, we're reimagining how Private Equity and consulting teams work—by building intelligent AI-native workflows and agentic platforms tailored to the real-world needs of high-performance teams. Our mission is to accelerate value creation through cutting-edge AI, system design, and product craftsmanship. We're a...

  • Senior Backend Engineer

    2 weeks ago

    India Infinite Possibilities Full time

    About UsAt GenAI Labs IP India Pvt Ltd, we're reimagining how Private Equity and consulting teams work—by building intelligent AI-native workflows and agentic platforms tailored to the real-world needs of high-performance teams. Our mission is to accelerate value creation through cutting-edge AI, system design, and product craftsmanship. We're a...

  • Senior Backend Engineer

    4 weeks ago

    India EZAIX Full time

    We're building the next-generation communications analytics and automation platform—one that fuses deep telemetry, enterprise-scale voice/calling data, and AI-driven remediation.As a Senior Backend Engineer, you'll play a core role in designing the resilient, scalable backend of a high-visibility platform that already drives action across global Microsoft...

  • Senior Backend Python Engineer 32426

    3 weeks ago

    India Stealth AI Startup Full time

    Compensation: Up to $300,000 USD annually + Equity Type: Full-time | Remote We're looking for a Senior Python Backend Engineer to help build infrastructure powering real-time AI workflows and ML pipelines.You'll work on high-availability systems and collaborate with a global team of engineers and researchers.Responsibilities: Build and optimize backend APIs...

  • Senior Backend Engineer

    4 weeks ago

    India Cloud Packets Full time

    Job DescriptionWe are looking for a Senior Backend Engineer with 8+ years of experience in C#/.NET and AWS serverless technologies to join our team This role is ideal for someone with expertise in Lambda, API Gateway, DynamoDB, SQS/SNS, and S3, who is passionate about building scalable, high-performance backend systems. You will play a key role in designing...

  • Backend Engineer

    4 weeks ago

    India Grow Your Staff Full time

    Grow Your Staff is looking for a Backend Engineer for a Computer Software firm in Germany. The position is a full-time remote opportunity.The role will have excellent growth opportunities. You will work directly with the team based in Germany.Experience required: 5 years+ CTC: 15-20 LPALocation: RemoteResponsibilities:Develop and maintain backend services...