AI Agent Evaluation Analyst

1 week ago


Pune, Maharashtra, India Mindrift Full time ₹ 60,000 - ₹ 1,20,000 per year

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. 

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

Who we're looking for:

We're looking for curious and intellectually proactive contributors, the kind of person who double-checks assumptions and plays devil's advocate.

Are you comfortable with ambiguity and complexity? Does an async, remote, flexible opportunity sound exciting? Would you like to learn how modern AI systems are tested and evaluated?

This is a flexible, project-based opportunity well-suited for:

  • Analysts, researchers, or consultants with strong critical thinking skills.
  • Students (senior undergrads / grad students) looking for an intellectually interesting gig.
  • People open to a part-time and non-permanent opportunity.

About the project:

We're on the hunt for QAs for autonomous AI agents for a new project focused on validating and improving complex task structures, policy logic, and agent evaluation frameworks. Throughout the project, you'll have to balance quality assurance, research, and logical problem-solving. This project opportunity is ideal for people who enjoy looking at systems holistically and thinking through scenarios, implications, and edge cases.

You do not need a coding background, but you must be curious, intellectually rigorous, and capable of evaluating the soundness and consistency of complex setups. If you've ever excelled in things like consulting, CHGK, Olympiads, case solving, or systems thinking — you might be a great fit.

What you'll be doing:

  • Reviewing evaluation tasks and scenarios for logic, completeness, and realism.
  • Identifying inconsistencies, missing assumptions, or unclear decision points.
  • Helping define clear expected behaviors (gold standards) for AI agents.
  • Annotating cause-effect relationships, reasoning paths, and plausible alternatives.
  • Thinking through complex systems and policies as a human would to ensure agents are tested properly.
  • Working closely with QA, writers, or developers to suggest refinements or edge case coverage.

How to get started:

Apply to this post, qualify, and get the chance to contribute to a project aligned with your skills, on your own schedule. Shape the future of AI while building tools that benefit everyone.

Requirements
  • Excellent analytical thinking: Can reason about complex systems, scenarios, and logical implications.
  • Strong attention to detail: Can spot contradictions, ambiguities, and vague requirements.
  • Familiarity with structured data formats: Can read, not necessarily write JSON/YAML.
  • Can assess scenarios holistically: What's missing, what's unrealistic, what might break?
  • Good communication and clear writing (in English) to document your findings.

We also value applicants who have:

  • Experience with policy evaluation, logic puzzles, case studies, or structured scenario design.
  • Background in consulting, academia, olympiads (e.g. logic/math/informatics), or research.
  • Exposure to LLMs, prompt engineering, or AI-generated content.
  • Familiarity with QA or test-case thinking (edge cases, failure modes, "what could go wrong").
  • Some understanding of how scoring or evaluation works in agent testing (precision, coverage, etc.).
Benefits
  • Get paid for your expertise, withrates that can go up to $15/hour depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.

  • AI Agent Developer

    4 days ago


    Pune, Maharashtra, India Cyclotron Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Role: AI Agent Developer - Microsoft Copilot & Agentic AIJob Type: Full Time Employment (FTE)Number of positions: 1Experience: 5+ YearsLocation: India (Remote / Work from home)Shift Timing: 11 AM IST to 8 PM ISTMust Have Skills:Deep expertise in Microsoft AI technologies including Copilot Studio, Microsoft 365 Agent SDK, Azure AI Services, Azure AI Foundry...


  • Pune, Maharashtra, India Agivant Technologies Full time US$ 1,25,000 - US$ 1,75,000 per year

    We are seeking a highly skilled and visionary Technical Lead to spearhead the design, development, and deployment of agentic AI systems. This role demands deep technical expertise in AI architecture, LLMs, and agent frameworks, along with a strategic mindset to align AI initiatives with business goals. The ideal candidate will have hands-on experience with...


  • Pune, Maharashtra, India Invezza Technologies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About Invezza TechnologiesInvezza is a technology consulting and outsourced product development company. We believe in growing together and creating long-term relationships with the common purpose of delivering innovative solutions and cutting-edge digital experiences. We are technically creative, innovators with a deep passion for technology.Work location -...


  • Pune, Maharashtra, India BMC Software Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Role Overview:We are looking for a visionary Enterprise IT Architect to lead the transformation of our IT organization into an Agentic AI-driven powerhouse. This individual will architect modern, scalable, AI-native IT platforms that optimize operations, drive automation, and showcase how next-gen IT can innovate at scale. You will work across...


  • Pune, Maharashtra, India MNR Solutions Pvt. Ltd. Full time

    Description : Role : Agentic AI Developer Locations : Mumbai | Pune | Gurugram Experience : 5+ Years Are you passionate about shaping the future of AI with cutting-edge tools like LangChain, CrewAI, and OpenAI APIs We're looking for a talented Agentic AI Developer to join our growing team and build intelligent agent-based systems that push...

  • Agentic AI Engineer

    2 days ago


    Pune, Maharashtra, India o3 Technology Solutions Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Graduate Program Agentic AI Engineer (Intux)Location: PuneType: Full-time | Entry-level (01 year experience)Start: Rolling cohortsAbout O3 & IntuxO3 Technology Solutions builds Intux, our Agentic AI platform, which powers hundreds of intelligent agents across business functions like Sales, Service, HR, Operations, Risk, and Proposals. You'll work on real...

  • Agentic AI Engineer

    12 hours ago


    Pune, Maharashtra, India O3 Technology Solutions Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Graduate Program — Agentic AI Engineer (Intux)Location: PuneType: Full-time | Entry-level (0–1 year experience)Start: Rolling cohortsAbout O3 & IntuxO3 Technology Solutions builds Intux, our Agentic AI platform, which powers hundreds of intelligent agents across business functions like Sales, Service, HR, Operations, Risk, and Proposals. You'll work on...

  • AI Engineer

    6 days ago


    Pune, Maharashtra, India AgileWaters Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Description :Key Responsibilities : - Design, develop, and deploy AI/ML models and pipelines to support product features and internal automation needs. - Build and integrate agentic workflows that allow AI agents to operate across multiple systems, take actions, and orchestrate end-to-end processes within our SaaS platform. - Leverage AWS...


  • Pune, Maharashtra, India RSquareSoft Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Title: Agentic AI DeveloperLocation:PuneJob Type:Full-timeExperience:6 months–2 yearsAbout the RoleWe're looking for an Agentic AI Developer who can build intelligent, autonomous AI systems using modern frameworks and tooling. You'll design, integrate, and optimize AI workflows leveraging LLMs, embeddings, vector databases, and orchestration...

  • Agentic AI Developer

    2 weeks ago


    Pune, Maharashtra, India Capgemini Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you'd like, where you'll be supported and inspired by a collaborative community of colleagues around the world, and where you'll be able to reimagine what's possible. Join us and help the world's leading organizations unlock the value of technology and...