llm qa

3 days ago

Mumbai, Maharashtra, India Virtusa Full time ₹ 20,00,000 - ₹ 25,00,000 per year

Key Responsibilities

Test Planning & Execution

Design and execute comprehensive test plans for applications integrating LLMs and AWS services.

Validate data pipelines, model outputs, APIs, and user interfaces with a focus on LLM-driven features such as text generation, summarization, and classification.

LLM Quality & Evaluation

Develop test cases for LLM responses, ensuring accuracy, relevance, consistency, and compliance.

Evaluate prompt quality, model outputs, and fine-tuned model behavior using both manual and automated techniques.

Support RAG pipelines and validate integration with vector databases or knowledge sources.

Automation & Tooling

Build and maintain automated test suites for backend services and LLM workflows using tools like PyTest, Postman, Selenium, or Playwright.

Integrate tests into CI/CD pipelines (e.g., using GitHub Actions, Jenkins, or CodePipeline).

AWS Integration Testing

Test cloud-native services and workflows built with AWS tools such as:

Lambda, S3, API Gateway, Step Functions, Bedrock, SageMaker, CloudWatch

Validate infrastructure-as-code using CloudFormation or CDK in test environments.

Performance, Security & Compliance Testing

Conduct performance testing of LLM APIs (e.g., latency, throughput, model fallback behavior).

Ensure compliance with data privacy, audit, and security standards when using LLMs in production workflows.

Collaboration & Documentation

Work closely with engineers, data scientists, and product managers to define quality metrics for LLM use cases.

Document test strategies, bug reports, and QA process improvements.

Technical Skills

5+ years of experience in software or data quality assurance.

Hands-on experience testing applications built on AWS (Lambda, S3, API Gateway, CloudWatch, Step Functions, etc.).

Experience testing or validating LLM-powered features (e.g., OpenAI, Amazon Bedrock, HuggingFace, Claude).

Strong skills in test automation (e.g., PyTest, Selenium, REST API testing).

Good understanding of CI/CD and DevOps in AWS environments.

Experience with Python or JavaScript/TypeScript for test scripting.

Nice to Have

Familiarity with LLMOps, prompt testing frameworks, or prompt injection scenarios.

Experience with LangChain, RAG patterns, or vector databases (e.g., Pinecone, FAISS).

Exposure to model evaluation metrics (e.g., BLEU, ROUGE, factual consistency).

Basic knowledge of data pipelines, ETL, or analytics workflows.

Preferred Experience

Testing GenAI or AI-enhanced applications in real-world environments.

Ensuring ethical AI usage, data privacy, and compliance in AI workflows.

Familiarity with monitoring LLM behaviour in production and logging tools.

About Virtusa

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

  • QA Automation Engineer

    3 days ago

    Mumbai, Maharashtra, India Wärtsilä Corporation Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    QA Automation Engineer - Digital Port SolutionsWärtsilä is a world leader in marine technology and leading the industry's transformation towards a decarbonised and sustainable future. With the world's widest portfolio and service network, we – Wärtsilians - deliver efficient, safe and sustainable integrated products and solutions to enhance the business...

  • Business QA Analyst

    4 days ago

    Mumbai, Maharashtra, India Wärtsilä Corporation Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Business QA Analyst - Digital Port SolutionsWärtsilä is a world leader in marine technology and leading the industry's transformation towards a decarbonised and sustainable future. With the world's widest portfolio and service network, we – Wärtsilians - deliver efficient, safe and sustainable integrated products and solutions to enhance the business of...

  • DevOps - Manager

    2 days ago

    Mumbai, Maharashtra, India Zycus Infotech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About UsZycus, recognized by leading analyst firms in procurement technology, empowers teams to unlock deep value through its comprehensive Source-to-Pay (S2P) solutions. At the heart of our S2P solution is the Merlin Agentic Platform, which orchestrates intelligent AI agents to deliver simplified, efficient, and compliant processes.The Merlin Intake Agent...

  • Technical Content Writer Intern

    4 days ago

    Mumbai, Maharashtra, India commversion solutions Full time ₹ 1,50,000 per year

    CommVersion is looking for a creative and curious Content Writer & AI Prompt Intern to join our team at CommVersion. This internship offers hands-on experience in crafting compelling written content and designing effective AI prompts to train chatbots for real-world business use. You'll work closely with our Operations, QA, and Training teams to learn how to...

  • AI Agent Evaluation Analyst

    6 hours ago

    Mumbai, Maharashtra, India Mindrift Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English proficiency.At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of...

  • Senior Software developer

    7 days ago

    Mumbai, Maharashtra, India Kan Innovations Pvt Ltd Full time ₹ 78,20,000 - ₹ 87,60,000 per year

    Position: Sr Software Developer/ArchitectLocation: Mumbai (On-site)Experience: 4+ Years | Type: Full-TimeAbout UsWe're a rapidly growing med-tech company reimagining point-of-care diagnostics through intelligent software and custom hardware. Our integrated platform combines Python, Flask, PyQt, and Chromium-based UIs with real-time USB-connected devices to...

  • Chief Technical Officer/Product Lead

    1 week ago

    Mumbai, Maharashtra, India Antal TECH jobs Full time ₹ 12,00,000 - ₹ 30,00,000 per year

    We seek a strategic, hands‑on leader to bridge vision and execution. The CTO/Product Lead will architect a scalable, secure platform while shaping a product roadmap that delivers measurable value to users across the supply‑chain and mobility ecosystem.AI‑First / AI‑Assisted DevelopmentLead the adoption of AI‑assisted engineering and automation...

  • Data Engineer

    1 week ago

    Navi Mumbai, Maharashtra, India Neuiq Technologies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Role & responsibilitiesJob Title: Data EngineerExperience: 3 - 7 yearsLocation: HybridEmployment Type: Full-timeAbout the RoleWe are looking for a Data Engineer who will play a key role in designing, building, and maintaining robust data pipelines that power our AI-based analytics and dashboarding solutions. The candidate should have a strong foundation in...

  • Tech Lead

    6 days ago

    Navi Mumbai, Maharashtra, India The Fast Way Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Only applicants witha minimum of 4 years of relevant experiencewill be considered for this role. Candidates with less experience are kindly requestednot to apply.About UsThe Fast Way is a Digital Agency in Mumbai which provides Digital Marketing services, Web Designing and Web Development. The Fast Way is an IT and Digital Marketing solution provider. Our...

  • QA Automation Engineer

    4 days ago

    Mumbai Metropolitan Region, India Wärtsilä Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Wärtsilä is a world leader in marine technology and leading the industry's transformation towards a decarbonised and sustainable future. With the world's widest portfolio and service network, we – Wärtsilians - deliver efficient, safe and sustainable integrated products and solutions to enhance the business of our customers. The opportunities presented...

  • QA Engineer L2 –

    2 weeks ago

    Mumbai, India Haptik Full time

    What do we want to accomplish and why we need you?  Jio Haptik is an AI leader having pioneered AI-powered innovation since 2013. Reliance Jio Digital Services acquired Haptik in April 2019. Haptik currently leads India’s AI market having become the first to process 15 billion+ two-way conversations across 10+ channels and in 135 languages. Haptik is also...

  • Business QA Analyst

    1 week ago

    Mumbai, India Wärtsilä Full time

    Business QA Analyst - Digital Port Solutions Wärtsilä is a world leader in marine technology and leading the industry's transformation towards a decarbonised and sustainable future. With the world’s widest portfolio and service network, we – Wärtsilians - deliver efficient, safe and sustainable integrated products and solutions to enhance the...

  • Business QA Analyst

    5 days ago

    Mumbai Metropolitan Region, India Wärtsilä Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Wärtsilä is a world leader in marine technology and leading the industry's transformation towards a decarbonised and sustainable future. With the world's widest portfolio and service network, we – Wärtsilians - deliver efficient, safe and sustainable integrated products and solutions to enhance the business of our customers. The opportunities presented...

  • Quality Assurance Engineer

    1 week ago

    Mumbai, India HR Full time

    Description :- Min 7+ years of experience in software quality engineering & assurance.- Min 3+ years of Quality engineering experience in AI/ML projects- Solid understanding of and experience in Data engineering - Thorough understanding of AI, Machine Learning concepts, architecture and algorithms.- Good understanding of :1. LLM frameworks2. LLMs models...

  • Associate Architect

    3 weeks ago

    Mumbai, Thiruvananthapuram / Trivandrum, Bengaluru, India Quantiphi Analytics Solutions Full time

    Job Description - 6+ years of relevant hands-on technical experience implementing, and developing cloud ML solutions on AWS. - Hands-on experience on AWS Machine Learning services. Proven experience using AWS Sagemaker leveraging different types of data sources, Training jobs, real-time and batch Inference, and Processing Jobs. - Good Experience developing...

  • Quality Assurance Engineer

    6 days ago

    Mumbai, India TerraFinn Technologies, Mumbai Full time

    Company Name: TerraFinn TechnologiesClient: Fitch RatingsWebsite: 25+ lacsMode: Full-time, remoteJoining: ImmediateLead/Senior Quality AssuranceAbout the RoleWe are seeking a highly skilled Automation QA Engineer with expertise in AI-driven testing to join our quality engineering team. The ideal candidate will have strong experience in building automated...

  • ▷ (Immediate Start) Quality Assurance Engineer

    4 weeks ago

    Hyderabad, Mumbai, India Reuters Full time

    Job Description About the Role: In this opportunity as Quality Assurance Engineer , you will: - Delivers testing solutions for a single product and/or participates in complex projects - May lead straight forward projects that are not complex in nature - Operates with moderate level of guidance and direction - Immediate project or QA/Software Development...

  • Operations Specialist

    3 weeks ago

    Mumbai, India INSIDEA Full time

    About the company:At INSIDEA, we are a fully remote organisation, hiring top talents from across the globe to deliver exceptional digital marketing and HubSpot solutions. Our diverse services are designed to enhance your brand’s online presence and drive real, measurable growth. From marketing optimization and content creation to full-scale HubSpot...

  • Operations Specialist

    3 weeks ago

    Mumbai, India INSIDEA Full time

    About the company: At INSIDEA, we are a fully remote organisation, hiring top talents from across the globe to deliver exceptional digital marketing and HubSpot solutions. Our diverse services are designed to enhance your brand’s online presence and drive real, measurable growth. From marketing optimization and content creation to full-scale HubSpot...