AI Automation Test Engineer

Job Information

Date Opened

27/08/2025

Job Type

Permanent

Work Experience

5+ years

Industry

IT Services

Salary

10-15 LPA

City

Bangalore North

Province

Karnataka

Country

India

Postal Code

560052

Job Description

Position Overview

We are seeking a passionate AI Automation Test Engineer to join our innovative UK based stealth startup, in Bengaluru. This role offers an exciting opportunity to shape the testing strategy for cutting-edge AI products while working in a fast-paced, dynamic environment. You'll be instrumental in ensuring the quality and reliability of our AI-driven solutions through comprehensive automated testing frameworks.

Key Responsibilities

AI/ML Testing

• Design and implement automated testing frameworks specifically for AI/ML models and systems
• Develop test strategies for model performance, accuracy, bias detection, and robustness
• Create automated pipelines for data validation, model training verification, and inference testing
• Implement A/B testing frameworks for AI model comparison and evaluation

Test Automation Development

• Build and maintain end-to-end test automation suites using modern testing frameworks
• Develop API testing automation for microservices and ML model endpoints
• Create performance and load testing automation for AI inference systems
• Design automated regression testing for continuous integration/deployment pipelines

Quality Assurance

• Collaborate with AI/ML engineers to define testing requirements and acceptance criteria
• Establish quality gates and metrics for AI model deployment
• Conduct exploratory testing of AI features and user experiences
• Monitor and analyse test results, providing actionable insights to development teams

Infrastructure & DevOps

• Set up and maintain testing environments and test data management systems
• Integrate automated tests with CI/CD pipelines using tools like Jenkins, ArgoCD, or GitHub Actions
• Implement monitoring and alerting for test execution and system health
• Manage test data pipelines and synthetic data generation for AI model testing

Required Qualifications

Technical Skills

• Programming: Proficiency in Python, with experience in testing frameworks (pytest, unittest, Robot Framework)
• AI/ML Testing: Understanding of machine learning concepts, model evaluation metrics, and testing methodologies
• Automation Tools: Hands-on experience with Selenium, Appium, or similar web/mobile automation tools
• API Testing: Experience with REST/GraphQL API testing using tools like Postman, Newman, or requests library
• Version Control: Proficient with Git and collaborative development workflows

Experience Requirements

• 3-6 years of experience in test automation and quality assurance
• 1-2 years of experience testing AI/ML systems or data-driven applications
• Experience with cloud platforms (AWS, Azure, or GCP) and containerization (Docker, Kubernetes)
• Familiarity with databases (SQL and NoSQL) and data validation techniques

Preferred Qualifications

• Experience with MLOps tools and practices (MLflow, Kubeflow, or similar)
• Knowledge of performance testing tools (JMeter, Locust, or K6)
• Understanding of data science workflows and model lifecycle management
• Experience with monitoring tools (Prometheus, Grafana, ELK stack)
• Background in statistics or data analysis

What We Offer

Compensation & Benefits

• Competitive salary up to ₹15,00,000 per annum
• Performance-based bonuses and annual increments
• Comprehensive health insurance for you and your family

Professional Growth

• Opportunity to work with cutting-edge AI technology in stealth mode
• Direct impact on product development and company direction
• Mentorship from experienced AI and engineering leaders
• Learning budget for courses, conferences, and certifications

Work Environment

• Modern office space in Bengaluru with all necessary amenities
• Collaborative, innovation-driven culture
• Regular team events and knowledge-sharing sessions

About the Role

As an early-stage team member, you'll have the unique opportunity to build testing practices from the ground up, directly influence product quality, and grow with the company. This position is ideal for someone who thrives in ambiguous environments, enjoys solving complex technical challenges, and wants to be part of building something revolutionary in the AI space.

Note: Due to our stealth mode status, specific product details will be shared during the interview process with qualified candidates who sign appropriate confidentiality agreements.

Application Process

Interested candidates should submit their resume along with:

• Brief cover letter explaining interest in AI testing and startup environments
• Any relevant certifications or achievements in testing or AI/ML domains

We are an equal opportunity employer committed to diversity and inclusion. All qualified applicants will receive consideration regardless of race, gender, age, religion, sexual orientation, or disability status.