Quality Assurance Engineer, LLM

Join Delphi - Where Innovation meets transformation

At Delphi, we believe in creating an environment where our people thrive. Our
hybrid work model
empowers you to choose where you work—whether it's from the office, your home, or a mix of both—so you can prioritize what matters most. We are committed to supporting your personal goals, family, and overall well- being while driving transformative results for our clients.

We welcome exceptional talent from anywhere across the globe. Interviews and onboarding are conducted virtually, reflecting our digital-first mindset.

Rooted in the region, we specialize in delivering tailored, impactful solutions in
Data, Advanced Analytics and AI, Infrastructure, Cloud Security, and Application Modernization.
Whether it's enabling
predictive analytics
, transforming operations with automation, or driving customer engagement with intelligent platforms, we are the trusted partner for organizations ready to embrace a smarter, more efficient future.

About the Role: Sr Consultant - Quality Assurance

Location:
India (Remote)

As an
Sr Consultant - Quality Assurance (AI)
, you will play a critical role in ensuring the performance, reliability, and trustworthiness of advanced Large Language Models (LLMs), sub-models, and multi-agent systems. You will design and execute comprehensive evaluation strategies, leveraging both manual and automated testing to assess model accuracy, consistency, and robustness. With strong expertise in Python and synthetic data generation, you'll simulate real-world and edge-case scenarios to validate model behavior under diverse conditions. Collaborating closely with Data Science and Engineering teams, you will align QA frameworks with model development, utilize cutting-edge evaluation tools and metrics, and maintain detailed documentation and prompt libraries. This role offers an exciting opportunity to shape the quality and performance standards of next-generation AI systems.

What You'll Do:

•       Design and execute
evaluation strategies for LLMs
, sub-models, and multi-agent systems.

•       Perform
manual and automated testing
of AI models, ensuring comprehensive coverage.

•       Develop
synthetic test data using Python scripting
to simulate real-world scenarios and edge cases.

•       Ensure
boundary condition coverage
through automated data generation techniques.

•       Prepare and maintain
prompt libraries
for model evaluation across diverse use cases.

•       Evaluate
multi-model and multi-agent architectures
, ensuring functional and performance benchmarks are met.

•       Apply and interpret
evaluation metrics
(e.g., accuracy, precision, recall, F1-score, Bias check, hallucination, Consistency, Relevance, Fact Check, BLEU, ROUGE, perplexity, etc.) with a clear understanding of their formulas and implications.

•       Collaborate closely with Data Science and Engineering teams to align QA strategies with model development.

•       Document test plans, test cases, and evaluation reports for stakeholder review and signoff.

What You'll Bring:

• years of relevant experience in quality assurance and AI/ML evaluation.

•       Strong hands-on experience with
LLM evaluation techniques
.

•       Proficiency in
Python
for scripting, test automation, and synthetic data generation.

•       Deep understanding of
AI model architecture
, including models, sub-models, and agent-based systems.

•       Familiarity with
prompt engineering
and its role in model testing.

•       Experience with
AI/ML testing frameworks
and tools.

•       Solid grasp of
evaluation metrics
and their practical application.

•       Excellent analytical, documentation, and communication skills.

•       Prior experience in QA for AI/ML products.

•       Exposure to tools like RAGAS, DeepEval, Arize Phoenix, OpenAI APIs, or similar.

•       Experience with CI/CD pipelines for AI model deployment and testing.

•       A sense of humor

What We Offer:

At Delphi, we are dedicated to creating an environment where you can thrive, both professionally and personally. Our
competitive compensation package, performance-based incentives,
and health benefits are designed to ensure you're well-supported. We believe in your continuous growth and offer
company- sponsored certifications, training programs
, and skill-building opportunities to help you succeed. We foster a culture of inclusivity and support, with
remote work
options and a fully supported work-from- home setup to ensure your comfort and productivity. Our positive and inclusive culture includes team activities, wellness and mental health programs to ensure you feel supported.