QA Architect I

9 - 12 Years

1 Opening

Trivandrum

Role description

Job Title

QE Architect – AI / LLM Systems

Role Summary

We are looking for a visionary QE Architect – AI/LLM Systems to architect, define, and drive the overall Quality Engineering (QE) strategy for next-generation AI products. This role will focus on building scalable quality architectures, AI evaluation frameworks, and automated testing pipelines to ensure reliable, safe, and high-quality AI-driven user experiences.

The ideal candidate will bring strong thought leadership and deep technical expertise, working at the intersection of AI/ML, LLM systems, software engineering, and quality governance.

Key Responsibilities

1. QE Architecture & Strategy

• Define and own the end-to-end quality architecture for all AI and LLM initiatives across the organization.

• Design enterprise-level QE frameworks and reusable components for:

• Conversational AI applications and chatbots

• Knowledge-management bots and RAG systems
• Semantic and vector-based text search
• Image search and multimodal AI systems
• Generative AI platforms
• Establish scalable testing pipelines for model evaluation, data validation, and automation.

2. AI / LLM Evaluation Frameworks

• Architect comprehensive evaluation systems for:

• Prompt testing and scenario-based validation

• LLM output quality, safety, bias, and consistency
• Hallucination detection and mitigation
• RAG correctness, grounding accuracy, and knowledge integrity
• Search relevance and ranking metrics
• Build automated scorecards and continuous evaluation dashboards.

3. Automation & Infrastructure

• Design and implement automation frameworks for:

• LLM APIs and chat agents

• Multimodal AI pipelines
• Vector databases and semantic search services

• Architect model regression detection using:

• Golden datasets

• Synthetic test data generation
• LLM-as-a-Judge approaches
• Self-evaluation and multi-agent evaluation techniques
• Integrate AI test harnesses into CI/CD and LLMOps pipelines.

4. Data Quality & Test Data Strategy

• Define enterprise-wide AI test data management strategies, including:

• Ground-truth datasets

• Benchmark datasets
• Adversarial and edge-case inputs
• Safety and compliance-focused test scenarios

5. Architecture Reviews & Cross-Team Leadership

• Provide architectural guidance to ML engineers, data engineers, and software teams on testability and observability.
• Review AI system architectures, including model pipelines, chatflows, orchestration layers, and search systems.
• Drive quality gates across experimentation, pre-production, and production rollout cycles.

6. Quality Governance & Best Practices

• Establish enterprise standards for:

• AI testing taxonomies and methodologies

• Privacy, safety, and compliance validation
• Defect classification for LLM-specific issues
• Reliability, latency, and scalability benchmarks
• Lead adoption of AI/ML Quality Engineering best practices across teams.

Required Qualifications

• 10+ years of experience in Quality Engineering, with at least 3+ years in AI/ML/LLM systems.

• Strong understanding of:

• Large Language Models (LLMs), NLP, embeddings, and vector databases

• Chatbot platforms such as Dialogflow, Rasa, Botpress, Amazon Lex, etc.
• RAG pipelines and knowledge-management systems
• Image search and multimodal AI architectures
• Strong programming experience in Python, Java, or TypeScript, with ML/NLP libraries.
• Proven experience building CI/CD-integrated AI test automation frameworks.

• Hands-on knowledge of AI evaluation metrics such as:

• Perplexity, factuality, grounding score

• CER/WER, BLEU, ROUGE
• MRR, NDCG, search relevance metrics
• Model drift and performance stability metrics
• Experience handling non-deterministic testing, probabilistic evaluation, and AI quality challenges.
• Proven ability to architect and scale enterprise-grade QE systems.

Core Skills

AI / ML / LLM Systems
• QE Architecture
• Python / Java / TypeScript
• CI/CD & LLMOps
• Automation Frameworks
• RAG & Vector Search
• AI Quality Metrics

Skills

Automation Testing, AI/ML/LLM system,python, Java or TypeScript,CI/CD, QE Architecture

About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world's best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients' organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.

---