AI - QA Engineer

About the job

What makes Techjays an inspiring place to work

At Techjays, we are driving the future of artificial intelligence with a bold mission to empower businesses worldwide by helping them build AI solutions that transform industries. As an established leader in the AI space, we combine deep expertise with a collaborative, agile approach to deliver impactful technology that drives meaningful change.

Our global team consists of professionals who have honed their skills at leading companies such as Google, Akamai, NetApp, ADP, Cognizant Consulting, and Capgemini. With engineering teams across the globe, we deliver tailored AI software and services to clients ranging from startups to large-scale enterprises.

Be part of a company that's pushing the boundaries of digital transformation. At Techjays, you'll work on exciting projects that redefine industries, innovate with the latest technologies, and contribute to solutions that make a real-world impact. Join us on our journey to shape the future with AI.

We are looking for a detail-oriented and curious AI QA Engineer to join our growing QA team. You will play a critical role in ensuring the quality, safety, and reliability of our AI-powered products and features. If you're passionate about AI, testing complex systems, and driving high standards of quality—this role is for you

Primary Skills: QA Automation, Python, API Testing, AI/ML Testing, Prompt Evaluation, Adversarial Testing, Risk-Based Testing, LLM-as-a-Judge, Model Metrics Validation, Test Strategy.

Secondary Skills: CI/CD Integration, Git, Cloud Platforms (AWS/GCP/Azure ML), MLFlow, Postman, Testim, Applitools, Collaboration Tools (Jira, Confluence), Synthetic Data Generation, AI Ethics & Bias Awareness.

Experience: 3 - 8 Years

Work Location: Coimbatore/ Remote

Must-Have Skills:

Foundational QA Skills

• Strong knowledge of test design, defect management, and QA lifecycle.
• Experience with risk-based testing and QA strategy.

AI/ML Knowledge

• Basic understanding of machine learning workflows, training/inference cycles.
• Awareness of AI quality challenges: bias, fairness, transparency.
• Familiarity with AI evaluation metrics: accuracy, precision, recall, F1-score.
• Hands-on with prompt testing, synthetic data generation, and non-deterministic behavior validation.

Technical Capabilities

• Python programming for test automation and data validation.
• Hands-on experience with API testing tools (Postman, Swagger, REST clients).
• Knowledge of test automation tools (e.g., PyTest, Playwright, Selenium).
• Familiarity with Git and version control best practices.
• Understanding of CI/CD pipelines and integration testing.

Tooling (Preferred)

• Tools like Diffblue, Testim, Applitools, Kolena, Galileo, MLFlow, Weights & Biases.
• Basic understanding of cloud-based AI platforms (AWS Sagemaker, Azure ML, GCP Vertex AI).

Soft Skills

• Excellent analytical thinking and attention to detail.
• Strong collaboration and communication skills to work across cross-functional teams.
• Proactive and pull-mode work ethic—self-starter who takes ownership.
• Passion for learning new technologies and contributing to AI quality practices.

Roles & Responsibilities:

• Design, write, and execute test plans and test cases for AI/ML-based applications.
• Collaborate with data scientists, ML engineers, and developers to understand model behavior and expected outcomes.
• Perform functional, regression, and exploratory testing on AI components and APIs.
• Validate model outputs for accuracy, fairness, bias, and explainability.
• Implement and run adversarial testing, edge cases, and out-of-distribution data scenarios.
• Conduct prompt testing and evaluation for LLM (Large Language Model)-based applications.
• Use LLM-as-a-Judge and AI tools to automate evaluation of AI responses where possible.
• Validate data pipelines, datasets, and ETL workflows.
• Track model performance metrics such as precision, recall, F1-score, and flag potential degradation.
• Document defects, inconsistencies, and raise risks proactively with the team.

What we offer:

• Best in packages
• Paid holidays and flexible paid time away
• Casual dress code & flexible working environment
• Medical Insurance covering self & family up to 4 lakhs per person.
• Work in an engaging, fast-paced environment with ample opportunities for professional development.
• Diverse and multicultural work environment
• Be part of an innovation-driven culture that provides the support and resources needed to succeed.