AI Evals

2 weeks ago

Mumbai, Maharashtra, India BharatGen Full time

Job Summary:

We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and customer-centricity, and are passionate about shaping the behavior of AI systems in the real world.

Key Responsibilities:

  • Build and maintain AI evaluation pipelines to test, measure, and evaluate the behavior and performance of AI systems.
  • Implement traces, spans, and session tracking for observability and identify error propagation in multi-step pipelines.
  • Define AI quality metrics and KPIs around factuality, faithfulness, toxicity, grounding precision/recall, latency, cost, etc., with clear acceptance bars.
  • Implement evaluation and testing automation to enable end-to-end system and regression testing at scale.
  • Define criteria for and implement release gates in the CI/CD pipeline.
  • Find creative ways to break products.
  • Assist in root cause analysis and troubleshooting of bugs and field issues.
  • Collaborate with cross-functional teammates from product, engineering, linguistics, and customer support to shape human-AI interaction paradigms and ensure that our AI models and applications deliver the desired outcome and user experience.

Minimum Qualifications and Experience:

  • Bachelor's or Master's degree in CS/CE/IT/EE/E&TC or related fields with 5+ years of experience in manual and automation testing of software products, with at least 2 years in evaluating and testing AI/ML products.

Required Expertise:

  • Strong software testing fundamentals and expertise in writing test plans, executing test cases, and generating detailed reports and dashboards.
  • Strong analytical and debugging skills, and attention to detail.
  • Proficiency in Python, scripting, and software testing automation frameworks and tools such as Pytest, Selenium, Robot Framework, etc.
  • Working knowledge of generative AI models, AI agents, and related concepts such as retrieval augmented generation (RAG), prompt engineering, context engineering, explainability, traceability, observability, guard rails, reasoning, specificity, etc.
  • Sound understanding of the fundamental differences in the approach for testing conventional software versus evaluating generative AI systems.
  • Team player with excellent interpersonal skills and the ability to collaborate effectively with remote and cross-functional team members.
  • Go-getter attitude and ability to flourish in a fast-paced, startup environment.

  • Experience in any of the following would be a big plus -

  • AI evaluation frameworks such as Arize, Braintrust, DeepEval, LangSmith, Ragas

  • AI safety and red teaming experience, e.g., prompt injection, jailbreak, adversarial and stress testing.

  • Different types of AI evaluation methods, e.g, Human-in-the-loop, LLM-as-a-Judge.

  • AI Jewelry Designer

    2 weeks ago

    Mumbai, Maharashtra, India Bez - The AI Copilot for Jewellers Full time

    About BezBez builds domain-specific AI for jewellery businesses. Our agents learn a brand's unique style, then generate original designs, manufacturing models, and marketing visuals in minutes—shrinking the product lifecycle by 5×. We work with some of the world's largest jewellery manufacturers and are backed by LeapYear & Macroscopic Ventures.Why This...

  • Forward Deployed AI Engineer

    7 days ago

    Mumbai, Maharashtra, India Weekday AI Full time ₹ 4,00,000 - ₹ 8,00,000 per year

    This role is for one of the Weekday's clientsSalary range: Rs Rs ie INR 8-15 LPA)Min Experience: 2 yearsLocation: Gujarat, Mumbai, PuneJobType: full-timeWe're building the future of agentic commerce — a low/no-code framework that helps enterprises create multimodal AI agents with observability, governance, and real-world customer context built in. As a...

  • AI Engineer

    5 days ago

    Mumbai, Maharashtra, India Newfold Digital Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About Us:Newfold Digital is a leading web technology company serving nearly seven million customers globally. Established in 2021 through the combination of leading web services providers Endurance Web Presence and Group, our portfolio of brands includes: BlueHost, CrazyDomains, HostGator, Network Solutions, , , and many others. We help customers of all...

  • Senior AI Engineer

    6 days ago

    Mumbai, Maharashtra, India NTT DATA Global Delivery Services Ltd Full time ₹ 15,00,000 - ₹ 30,00,000 per year

    Senior GenAI Engineers Req ID: 341050 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Senior GenAI Engineers to join our team in Mumbai, Mahārāshtra (IN-MH), India (IN). ...

  • Founding Head of Engineering

    2 weeks ago

    Mumbai, Maharashtra, India Propkee Full time US$ 1,04,000 - US$ 1,30,878 per year

    We're an ambitious, AI‑native company in prop commerce. We move fast, build pragmatically, and care about craft. This role is for atech maverickwho prefers code over calendar invites and prototypes before pontificating.Location:MumbaiType:Full‑timeRole:Founding, hands‑on IC (no people management to start)Why this role exists (and why now)We're entering...

  • Senior Machine Learning Engineer

    2 weeks ago

    Mumbai, Maharashtra, India ClanX Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    OverviewHands-on Machine Learning Engineer to lead and scale ML infrastructure, models, and pipelines for agentic AI products at Nova.Company detailsNova builds AI teammates to help businesses protect revenue and automate manual tasks in finance, legal, and compliance.Website: Founder: Requirements4–5 years of experience as an ML Engineer or Applied...

  • Head of Engineering

    2 weeks ago

    Mumbai, Maharashtra, India Propkee Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    About the RoleWe're an ambitious, AI-native company in prop-commerce. The next 12–18 months are about turning hard, ambiguous problems into robust systems with real users. We're looking for a Founding Engineer who thrives on building end-to-end: designing data models, writing production code, and shipping fast.This is a hands-on IC role (no people...

  • GenAI Engineer

    2 weeks ago

    Mumbai, Maharashtra, India NTT DATA Global Delivery Services Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Mid level GenAI Engineers Req ID: 341095 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Mid level GenAI Engineers to join our team in Mumbai, Mahārāshtra (IN-MH), India...

  • Mid level GenAI Engineers

    6 days ago

    Mumbai, Maharashtra, India NTT DATA Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Req ID: 341095NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Mid level GenAI Engineers to join our team in Mumbai, Mahārāshtra (IN-MH), India (IN). Key Responsibilities Build...

  • Ai evals

    2 weeks ago

    Mumbai, India BharatGen Full time

    Job Summary:We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and...

  • Ai evals

    1 week ago

    Mumbai, India BharatGen Full time

    Job Summary:We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and...

  • AI Jewelry Designer

    2 weeks ago

    Mumbai, India Bez - The AI Copilot for Jewellers Full time

    About Bez Bez builds domain-specific AI for jewellery businesses. Our agents learn a brand’s unique style, then generate original designs, manufacturing models, and marketing visuals in minutes—shrinking the product lifecycle by 5×. We work with some of the world's largest jewellery manufacturers and are backed by LeapYear & Macroscopic Ventures. Why...

  • AI Jewelry Designer

    2 weeks ago

    Mumbai, India Bez - The AI Copilot for Jewellers Full time

    About Bez Bez builds domain-specific AI for jewellery businesses. Our agents learn a brand’s unique style, then generate original designs, manufacturing models, and marketing visuals in minutes—shrinking the product lifecycle by 5×. We work with some of the world's largest jewellery manufacturers and are backed by LeapYear & Macroscopic Ventures. Why...

  • AI Jewelry Designer

    2 weeks ago

    Mumbai, India Bez - The AI Copilot for Jewellers Full time

    About BezBez builds domain-specific AI for jewellery businesses. Our agents learn a brand’s unique style, then generate original designs, manufacturing models, and marketing visuals in minutes—shrinking the product lifecycle by 5×. We work with some of the world's largest jewellery manufacturers and are backed by LeapYear & Macroscopic Ventures.Why This...

  • AI Jewelry Designer

    2 weeks ago

    Mumbai, India Bez - The AI Copilot for Jewellers Full time

    About Bez Bez builds domain-specific AI for jewellery businesses. Our agents learn a brand’s unique style, then generate original designs, manufacturing models, and marketing visuals in minutes—shrinking the product lifecycle by 5×. We work with some of the world's largest jewellery manufacturers and are backed by LeapYear & Macroscopic Ventures. Why...

  • AI Evals

    2 weeks ago

    Mumbai, India BharatGen Full time

    Job Summary:We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and...

  • AI Evals

    2 weeks ago

    mumbai, India BharatGen Full time

    Job Summary:We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and...

  • AI Evals

    2 weeks ago

    Mumbai, India BharatGen Full time

    Job Summary:We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and...

  • AI Evals

    2 weeks ago

    Mumbai, India BharatGen Full time

    Job Summary:We are looking for an AI Evaluation & Test Engineer to join our growing team to ensure that our generative AI models and applications are safe, accurate, trustworthy, and deliver an elegant user experience. You will serve as the first customer of our AI systems. This role is ideal for product-minded engineers who obsess over product quality and...