Gen AI QA Engineer – AI Consulting

Experience : 5+ Years Experience

Education : Bachelor's/Master's degree

Work Location : Chennai, India (Chennai/Remote/Hybrid)

Key Responsibilities :

• Design, develop, and maintain automated testing frameworks for Gen AI applications and agentic systems.
• Implement evaluation pipelines to measure and improve LLM reliability and response quality.
• Develop and operationalize hallucination and bias detection systems.
• Collaborate with LLM Engineers to integrate evaluation checkpoints into model deployment workflows.
• Define and manage evaluation datasets and metrics for automated quality reporting.
• Partner with Product and Governance teams to ensure responsible AI testing and validation practices.
• Continuously monitor and improve AI performance, reliability, and trust metrics.

Required Technical Skills :

Generative AI Quality Engineering

• Design automated testing frameworks for generative AI and multi-agent systems.
• Develop evaluation pipelines to assess LLM responses for accuracy, coherence, and factual grounding.
• Implement hallucination detection mechanisms using reference comparison, confidence scoring, or retrieval checks.
• Create reproducible test datasets and evaluation benchmarks for prompt-response validation.

Automation Frameworks & Tools

• Develop and maintain Python-based QA frameworks integrated with CI/CD systems.
• Experience with testing tools such as pytest, LangSmith, DeepEval, or TruLens.
• Automate testing of API-driven AI systems, RAG pipelines, and multi-agent conversations.
• Implement regression, performance, and stress testing for large-scale AI applications.

Evaluation Metrics & Analysis

• Define and compute LLM evaluation metrics (precision, recall, coherence, diversity, factual accuracy, toxicity).
• Build evaluation dashboards and reporting tools for continuous model monitoring.
• Analyze model drift, prompt sensitivity, and performance degradation over time.

Model Safety & Reliability

• Develop and integrate toxicity, bias, and compliance detection checks into QA pipelines.
• Collaborate with security and governance teams to ensure AI safety and alignment standards.
• Implement guardrail testing and policy compliance frameworks for generative systems.

Infrastructure & Integration

• Integrate QA pipelines into CI/CD workflows for LLM deployment pipelines.
• Experience with cloud-native tools (AWS, Azure, GCP, or OCI) for automated test execution.
• Implement test orchestration using Docker, Kubernetes, or serverless workflows.
• Familiarity with API mocking, synthetic test data generation, and version-controlled evaluation artifacts.

Required Skills & Experience :

• 5+ years of experience in QA Automation, AI Testing, or ML Evaluation Engineering.
• Strong programming background in Python with experience in PyTest, FastAPI, or Playwright.
• Understanding of LLM architectures, prompt-response mechanics, and RAG systems.
• Experience with AI evaluation frameworks such as DeepEval, TruLens, or LangSmith.
• Familiarity with data validation, model observability, and pipeline testing.
• Proven ability to design automated evaluation frameworks for AI and ML models.

APPLY

Close

Drivestream's Employee Benefits.
Remuneration

Drivestream offers competitive pay and attracts a diverse community of skilled individuals. We recognize the value of investing in our talent.

Medical, Disability and Life Insurance

We provide an array of coverage options including full medical, full dental and vision plans, employee life insurance, LTD and STD coverage, flexible spending account and employee accidental death and dismemberment

Leave Benefits

Drivestream's generous paid leave programs feature vacation/paid time off (PTO), holiday leave and bereavement leave

Professional Development

Our training and development programs include traditional classroom training, online courses, including leadership, communication and project planning development, strategic planning and management programs, and professional society membership incentive

Work/Life Programs

Drivestream offers Work-Life Integration options that help individuals manage their personal and professional responsibilities. Options Include work from home and telecommuting, day care, flexible spending accounts, internal job transfer, and career mobility, and health and wellness programs

Community Involvement

Drivestream believes in supporting community and philanthropic activities that allow our employees to engage in outreach and educational programs.

Awards Programs

Drivestream recognizes and rewards our staff through various annual awards programs.

Retirement Benefits

Drivestream offers complete 401(k) plans and annual profit-sharing contribution.

---