Principal Tester

Lead QA – Automated TestingThe role of Lead Quality Assurance in automated testing & AI validation involves overseeing quality assurance initiatives for cutting-edge generative AI and multi-agent systems. The ideal candidate will have strong expertise in Python automation, LLM evaluation, RAG pipelines, observability, adversarial testing, and Azure monitoring.Design and execute LLM evaluation frameworks using LLM-as-a-Judge (G-Eval, custom evaluators) and hallucination detection, faithfulness, relevance, precision/recall metrics.Implement RAG evaluation frameworks (RAGAS or similar).Build Python-based automation frameworks using PyTest & DeepEval.Integrate automation into CI/CD pipelines using GitHub Actions.Perform adversarial and red-team testing: prompt injection, jailbreak attacks, bias and toxicity detection, and API testing for microservices (REST, async workflows).Required Skills:Test Automation: Python, PyTest, DeepEval.LLM Evaluation: G-Eval, Custom Evaluators, LLM-as-a-Judge.RAG Evaluation: RAGAS, Retrieval Metrics.Evaluation Metrics: Hallucination, Faithfulness, Relevance, Precision/Recall.Observability & Monitoring: LangFuse, LangSmith.CI/CD: GitHub Actions.Multi-Agent Testing: Reasoning & Tool Validation.Adversarial/Red Team Testing: Prompt Injection, Jailbreak, Bias/Toxicity.API Testing: REST & Async Workflows.Azure Monitoring: App Insights, Log Analytics.Synthetic & Golden Dataset Management.Automated Scoring System Design for GenAI Outputs.