AI Quality Assurance Expert

1 day ago

Bengaluru, Karnataka, India beBeeQuality Full time ₹ 1,75,00,000 - ₹ 2,50,00,000
Job Overview">

Leading Quality Assurance for Generative AI Solutions: An Exciting Opportunity to Drive Excellence

Key Responsibilities:
  1. Design and implement comprehensive end-to-end QA strategies for applications leveraging Node.js, integrated with LLMs, retrieval-augmented generation (RAG), and Agentic AI workflows.
  2. Establish benchmarks and quality metrics for GenAI components including accuracy, coherence, relevance, stability, and safety.
  3. Develop structured evaluation datasheets for LLM behavior validation: test prompts, expected responses, classification criteria, and scoring rubrics.
  4. Perform data quality testing for RAG databases and ensure relevant, high-quality retrieval to minimize hallucinations and improve grounding.
  5. Conduct A/B testing across model versions, prompt designs, and system configurations to measure and compare output quality.
  6. Define methodologies and simulate non-deterministic behaviors using Agentic AI testing techniques.
  7. Closely collaborate with developers, product owners, and AI engineers to test prompt engineering pipelines, function-calling interfaces, and fallback logic.
  8. Build QA automation where applicable and integrate GenAI evaluations into CI/CD pipelines.
  9. Lead internal capability development by mentoring QA peers on GenAI testing practices and helping evolve the organization's AI quality standards.
Requirements:
  • 6+ years of experience in software quality assurance, with at least 3+ years working in or around GenAI or LLM-based systems.
  • Deep understanding of GenAI quality dimensions: response grounding, factual correctness, context awareness, and hallucination minimization.
  • Experience creating and maintaining LLM evaluation datasets and designing test cases for dynamic prompt behavior.
  • Hands-on experience with tools and techniques for testing retrieval pipelines, embedding quality, and vector similarity results in RAG architectures.
  • Familiarity with non-deterministic testing strategies, agent loop evaluation, and multi-step LLM task validation.
  • Comfortable working with APIs, logs, test scripts, and tracing tools to validate both system and AI behavior.
  • Strong analytical thinking and a methodical approach to identifying bugs, regressions, and inconsistencies in AI outputs.
Desirable Skills:
  • Experience with GenAI tools/platforms like OpenAI, LangChain, Semantic Kernel, Hugging Face, Pinecone, or Weaviate.
  • Exposure to evaluating LLMs in production settings, including safety nets, guardrails, and red-teaming approaches.
  • Familiarity with prompt tuning, few-shot learning, and function/tool calling in LLMs.
  • Basic scripting knowledge (Python, JavaScript, or TypeScript) for building test harnesses or validation utilities.

  • Experienced Quality Assurance Expert Wanted for Strategic Testing Leadership

    1 day ago

    Bengaluru, Karnataka, India beBeeQuality Assurance Full time ₹ 2,00,00,000 - ₹ 3,00,00,000

    Job OpportunityWe are seeking a seasoned quality assurance expert to lead our testing efforts and drive excellence in software development.The ideal candidate will have a strong track record of quality leadership, strategic planning, and team management.This role is responsible for defining and implementing comprehensive QA strategies, test methodologies,...

  • AI Quality Assurance Manager

    3 days ago

    Bengaluru, Karnataka, India beBeeai Full time ₹ 18,31,400 - ₹ 24,91,900

    AI Quality Assurance ManagerWe're looking for a seasoned AI quality assurance professional to lead our team and drive excellence in conversational AI solutions.

  • - Quality Assurance Architect

    2 days ago

    Bengaluru, Karnataka, India Skan ai Full time

    We are seeking an experienced QA Architect with a proven track record of excellence in quality assurance leadership and strategic testing ideal candidate will have a minimum of 12 years of hands-on experience in software testing with strong expertise in test automation, architecture design, and team management.This role primarily focuses on strategic QA...

  • AI Quality Assurance Engineer

    1 day ago

    Bengaluru, Karnataka, India beBeeQuality Full time ₹ 2,20,00,000 - ₹ 2,54,80,000

    Job Overview">We are seeking a skilled AI Quality Assurance Engineer to join our team. As an AI Quality Assurance Engineer, you will play a critical role in ensuring the quality, performance, and security of our Core AI Services.">About the Role">">You will be responsible for validating distributed, cloud-native services and public APIs that form the...

  • AI Detective: Quality Engineering Expert

    3 days ago

    Bengaluru, Karnataka, India beBeeInnovative Full time ₹ 2,00,00,000 - ₹ 3,00,00,000

    About UsJob Title:IDetective, AI Architect with a strong background in Quality Engineering to lead the design and implementation of intelligent systems that enhance software quality, testing automation, and delivery efficiency.We are seeking an expert with deep expertise in AI/ML technologies combined with hands-on experience in quality assurance, test...

  • AI and Machine Learning Quality Assurance Specialist

    7 days ago

    Bengaluru, Karnataka, India beBeeOptimization Full time ₹ 18,00,000 - ₹ 25,50,000

    Job OpportunityWe're seeking an AI and Machine Learning Quality Assurance Specialist with expertise in testing artificial intelligence models for customer-facing applications, particularly Salesforce. This role focuses on designing, evaluating, and optimizing AI prompts to ensure accurate results from these models.">Required Skills and QualificationsStrong...

  • Chief Quality Assurance Architect

    2 days ago

    Bengaluru, Karnataka, India beBeeQuality Full time ₹ 30,00,000 - ₹ 40,00,000

    Senior Quality Assurance StrategistWe are seeking a highly experienced professional to spearhead our quality assurance strategy, driving product excellence through expert leadership and innovative testing approaches.This role offers a unique opportunity to work on diverse projects, delivering high-quality solutions in a fast-paced environment.Main...

  • Software Quality Assurance Expert

    7 days ago

    Bengaluru, Karnataka, India beBeeSoftwareQualityAssuranceExpert Full time ₹ 8,00,000 - ₹ 12,00,000

    Job Title: Software Quality Assurance Expert At our organization, we're dedicated to creating a global Human Capital Management (HCM) platform. This cutting-edge platform is augmented with a suite of AI-powered applications to revolutionize talent management. As a key member of our team, you will collaborate closely with our Product team to understand use...

  • Expert Quality Assurance Specialist

    6 days ago

    Bengaluru, Karnataka, India beBeeQuality Full time

    Job Opportunity: Expert Quality Assurance SpecialistSeeking an experienced and skilled quality assurance specialist to join our team. This role involves rigorous testing, automation framework building, test plan execution, and documentation of test results for Salesforce applications.Key Responsibilities:• Perform manual and automated testing for...

  • Quality Assurance Automation Expert

    2 days ago

    Bengaluru, Karnataka, India beBeeQuality Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Quality Assurance Automation ExpertWe are seeking a highly skilled and experienced individual to lead our quality assurance automation efforts. The ideal candidate will have expertise in designing, developing, and maintaining scalable automation frameworks for web, API, and mobile applications.Key Responsibilities:Automation Framework...