QA Engineer III

2 weeks ago

Serilingampalli Telangana, India RealPage, Inc. Full time ₹ 6,00,000 - ₹ 18,00,000 per year

Overview
As a member of our Agentic AI Engineering team, we seek individuals passionate about advancing the quality and reliability of AI systems through rigorous evaluation, experimentation, and tooling. You will play a key role in assessing the performance, safety, and alignment of AI models, leveraging platforms like OpenAI and LangSmith. Our team thrives in a culture of curiosity, continuous learning, and collaborative innovation, where your contributions directly impact the trustworthiness of AI solutions used globally.

Responsibilities

  • Design and execute evaluation strategies for LLM-based systems using LangSmith, OpenAI tools, and custom frameworks.
  • Collaborate with product managers, researchers, and engineers to define evaluation metrics aligned with business and ethical goals.
  • Develop automated pipelines for regression testing, prompt evaluation, and model comparison.
  • Analyze model outputs for correctness, bias, safety, and hallucination using both qualitative and quantitative methods.
  • Build reusable test harnesses and datasets for benchmarking AI capabilities across tasks and domains.
  • Contribute to the development of internal tools for prompt engineering, evaluation visualization, and feedback loops.
  • Act as a quality advocate for AI-driven features across the product lifecycle.
  • Mentor junior team members in AI evaluation best practices and tooling.
  • Stay current with advancements in LLM evaluation, interpretability, and responsible AI.

Qualifications

  • Bachelor's or master's degree in computer science or a related technical field.
  • Minimum 6-8 years of experience into QA and Automation
  • Basic level understanding of software development lifecycle and quality best practices and tools
  • Good knowledge on Automation Framework Development
  • Proficient in the use of various automation frameworks like Playwright (preferred), Protractor, Rest Assured, karate etc.
  • Hands-on experience with OpenAI APIs (e.g., GPT-4, function calling, Assistants API).
  • Familiarity with OpenAI Evals and LangSmith for prompt orchestration and evaluation.
  • Strong Python skills, especially in building test automation and data analysis scripts.
  • Experience with prompt engineering, LLM fine-tuning, or RAG pipelines is a plus.
  • Proficiency in using tools like Jupyter, Pandas, and/or SQL for evaluation analytics.
  • Understanding of ethical AI principles, model alignment, and safety concerns.
  • Experience with version control (GitHub), CI/CD pipelines, and test management tools (e.g., JIRA, Azure Test Plans).
  • Excellent communication and collaboration skills, with the ability to explain complex AI behaviors to diverse stakeholders.

Bachelor's or master's degree in computer science or a related technical field.

  • QA Engineer

    3 days ago

    Serilingampalli, Telangana, India Vertis Full time ₹ 7,50,000 - ₹ 18,00,000 per year

    About Us:Innovate. Create. Elevate.At Vertis, we are on a mission to constantly innovate and create exceptional digital solutions. Our goal is to elevate our clients brands and the experiences they deliver to their customers through the power of technology-driven creativity. Our people are the catalyst, empowering our clients to successfully transform their...

  • QA Automation

    4 days ago

    Hyderabad, Telangana, India QA programmer Full time

    **ROLE :QA Automation (cypress)** **Location :Bangalore ,Hyderabad** **Experience:2+years** **Mandatory skills :Cypress** As a QA Automation Engineer specializing in Cypress, you will be responsible for designing, developing, and maintaining automated tests to ensure the high quality of our software products. You will work closely with cross-functional...

  • Cloud Engineer Iii

    2 weeks ago

    Hyderabad, Telangana, India Jobted IN C2 Full time

    About McDonalds in India One of the world s largest employers with locations in more than 100 countries McDonald s Corporation has corporate opportunities in Hyderabad Our global offices serve as dynamic innovation and operations hubs designed to expand McDonald s global talent base and in-house expertise Our new office in Hyderabad will bring together...

  • QA Engineer

    20 hours ago

    Hyderabad / Vizag, Telangana, , India Xtglobal Infotech Limited Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    QA EngineerHyderabad / Vizag - Hybrid work once a week from officeFull time 6+ years experience UK Shift Hours: 2 -11 PM IST Budget: LPA DOE Job Details: We are looking to hire a QA – Automation (Web  & Mobile testing)with primary testing skills on Selenium and Appium.Candidates need to have good communication skills and self -driven, committed and...

  • Sdet Iii

    3 weeks ago

    Hyderabad, Telangana, India Entain Full time

    Company Description Entain India is the engineering and delivery powerhouse for Entain one of the world s leading global sports and gaming groups Established in Hyderabad in 2001 we ve grown from a small tech hub into a dynamic force delivering cutting-edge software solutions and support services that power billions of transactions for millions of users...

  • QA Engineer

    2 weeks ago

    Hyderabad, Telangana, India NationsBenefits, LLC Full time

    Qualifications: - 7+ years in QA Automation, with 3+ years in a leadership role. - Experience with Software Development Life Cycle (SDLC), Software Testing Life Cycle (STLC), agile - model and different test methodologies & validation to ensure the Quality Assurance Control. - Experience using Relational Databases in Client/Server and Web environment -...

  • Qa Engineer

    3 weeks ago

    Hyderabad, Telangana, India Awign Expert Full time

    About the client - We are a world-changing team of AI researchers and engineers working on the cutting edge of generative AI We are building systems that work across telephony chat video email text to assist accelerate the human workforce with artificial agents Our focus is on helping customers improve their outcomes in the areas of Business Process Customer...

  • QA Engineer

    2 weeks ago

    HITEC City, Hyderabad, Telangana, India Vertis Digital Sdn Bhd Full time

    **About Us** **Innovate. Create. Elevate.** At Vertis, we are on a mission to constantly innovate and create exceptional digital solutions. Our goal is to elevate our clients brands and the experiences they deliver to their customers through the power of technology-driven creativity. Our people are the catalyst, empowering our clients to successfully...

  • QA Automation Engineer

    2 weeks ago

    Hyderabad, Telangana, India karmanya Full time

    **Job Overview**: We are looking for a QA Automation Engineer to join our team of IT professionals. In this role, you will be suggesting ideas for automated software test procedures. Besides, you will be highlighting problem areas and tracking bugs that hinder software installation. You will also be designing and installing software databases. As a QA...

  • QA Engineer

    14 hours ago

    Hybrid (Hyderabad, Telangana, IN) Aubrant Digitial Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About the roleAs a Quality Engineer on the Engineering team, you will participate in all aspects of the software development lifecycle which includes estimating, technical design, implementation, documentation, testing, deployment and support of our SaaS platform for our enterprise clients. You will be part of an agile team working with other engineers,...