Mid QA

Job Title: Mid QA – Automated Testing & AI ValidationExperience: 3+ YearsEmployment Type: Contract (Short Term)OverviewWe are seeking a Mid-Level QA Engineer – Automated Testing & AI Validation with hands-on experience in testing AI-powered systems, particularly LLM-driven applications. The ideal candidate will have a strong QA automation background, a growing understanding of probabilistic AI outputs, and enthusiasm for building and improving AI evaluation frameworks in real-world, production environments.This role requires high autonomy, comfort with ambiguity, and strong written communication skills in an async-first setup.Key Responsibilities- Design, develop, and execute automated test cases using Python and Pytest for AI-driven applications.- Validate LLM integrations, APIs, and multi-agent workflows through functional, regression, and smoke testing.- Perform intent classification, semantic similarity, and response consistency testing for conversational AI systems.- Conduct hallucination detection and factual accuracy checks using automated and semi-automated methods.- Implement response quality scoring using LLM-as-a-Judge evaluation patterns.- Use LLM observability and tracing tools such as LangFuse or LangSmith to monitor and validate model behavior.- Test conversational AI applications including chatbots and virtual assistants across use cases.- Support Kubernetes-based application health checks and basic smoke testing.- Integrate automated tests into CI/CD pipelines using GitHub Actions.- Document test cases, evaluation criteria, defects, and QA findings clearly and concisely.- Collaborate with engineering and AI teams to improve evaluation pipelines and testing strategies.- Continuously learn and contribute to advanced AI evaluation methodologies.Skills- 3–5 years of experience in QA or test automation roles.- Strong proficiency in Python-based test frameworks (Pytest).- Basic to intermediate understanding of LLM evaluation concepts and AI system testing.- Hands-on experience with LLM observability and tracing tools (LangFuse, LangSmith, or similar).- Experience in API testing and validation of AI integrations.- Exposure to conversational AI or chatbot testing.- Familiarity with CI/CD processes and GitHub Actions.- Knowledge of Kubernetes-based environments for testing and validation.- Strong documentation and defect reporting skills.- Ability to understand semantic correctness over strict assertion-based testing.- Self-driven mindset with the ability to work independently and handle ambiguity.Preferred Skills- Experience testing production-grade LLM systems with real user traffic.- Exposure to LLM-as-a-Judge frameworks and custom evaluation pipelines.- Familiarity with Azure-based infrastructure (AKS, Key Vault, PostgreSQL).- Understanding of multi-agent frameworks such as LangGraph or Microsoft Agent Framework.- Interest in AI ethics, responsible AI, and high-quality evaluation practices.