AI Evaluation Engineer

Role Summary

We are seeking an 
AI Evaluation Engineer
 to join our team and help define how next-generation AI systems are tested, trusted, and improved. In this role, you'll design and implement rigorous quality assurance and evaluation frameworks—combining automated pipelines, human-in-the-loop review, and synthetic data generation—to measure not only our platform reliability but also AI agents' accuracy, safety, and alignment with real-world use cases. You'll work end-to-end across the product lifecycle: writing test case scenarios, building automated tests, managing release test plans, developing dashboards and analysis tools, and translating insights into actionable improvements for both internal teams and clients.

Key Responsibilities

●       
Design evaluation frameworks
 for accuracy, safety, fairness, and alignment with intended use cases.

●       
Build and maintain evaluation pipelines
 that combine automated systems, human-in-the-loop review, and synthetic data generation to test AI Agents' performance at scale.

●       
Conduct failure mode and edge-case analysis
 to surface weaknesses, risks, and unexpected behaviors in AI outputs.

●       
Develop internal tools and dashboards
 that make evaluation results transparent, reproducible, and actionable across engineering, research, and client teams.

●       
Ensure evaluation datasets
 are diverse, representative, and high-quality, minimizing bias while capturing real-world complexity.

●       
Collaborate with researchers, engineers, and product stakeholders
 to translate insights into prioritized improvements and product decisions.

●       
Treat evaluation as a discipline of testing
—applying statistical rigor, reproducibility, and operational reliability across the AI lifecycle.

●       
Ensure deployment readiness
 by stress-testing agents for resilience, safety, and alignment in production-like environments.

●       
Quality Assurance
 ensure software is built to specifications. It is reliable, robust, secure, and ready for deployment. Create test cases, test plan, and bug reporting process for unit, regression, and UAT testing.

Qualifications & Skills

Required

●       Strong software engineering skills, with proficiency in Python and familiarity with data pipelines, APIs, and evaluation tooling.

●       Solid understanding of the machine learning lifecycle, including model training, testing, and deployment.

●       Experience designing or implementing evaluation metrics, experiment design, or statistical analysis.

●       Exposure to human-in-the-loop workflows, annotation systems, or synthetic data generation.

●       Ability to conduct rigorous failure analysis and translate results into actionable insights.

●       Clear, precise communication skills; able to present evaluation findings to technical and non-technical audiences.

Preferred

●       Quality Assurance and Testing Experience

●       Experience with LLMs, generative AI systems, or agentic workflows.

●       Familiarity with fairness, bias detection, interpretability, or safety evaluation.

●       Background in building dashboards, monitoring tools, or large-scale observability systems.

●       Prior work with evaluation frameworks, testing suites, or reproducibility practices at scale.

●       Comfort working end-to-end: from scoping evaluation goals to delivering deployment-ready results.

Seniority Levels / Variations

Depending on seniority (e.g., junior vs senior vs staff), responsibilities might scale to include:

●       Owning or leading an evaluation strategy at a product or platform level.

●       Mentoring others or managing QA teams

●       Architecture of evaluation platforms.

●       Setting standards for metrics, tools, and best practices across multiple product lines.

What We Offer / Why Join Us

●       Opportunity to influence AI product quality, fairness, and trust at scale.

●       Working with cutting-edge model architectures and AI tools.

●       Collaborating with top researchers / engineers / product leaders.

●       Flexibility / remote / collaborative environment (if applicable).

●       Learning opportunities in safety, fairness, interpretability, and evaluation methodologies.