QA Engineer III

1 week ago

Serilingampalli Telangana, India RealPage, Inc. Full time ₹ 6,00,000 - ₹ 18,00,000 per year

Overview
As a member of our Agentic AI Engineering team, we seek individuals passionate about advancing the quality and reliability of AI systems through rigorous evaluation, experimentation, and tooling. You will play a key role in assessing the performance, safety, and alignment of AI models, leveraging platforms like OpenAI and LangSmith. Our team thrives in a culture of curiosity, continuous learning, and collaborative innovation, where your contributions directly impact the trustworthiness of AI solutions used globally.

Responsibilities

  • Design and execute evaluation strategies for LLM-based systems using LangSmith, OpenAI tools, and custom frameworks.
  • Collaborate with product managers, researchers, and engineers to define evaluation metrics aligned with business and ethical goals.
  • Develop automated pipelines for regression testing, prompt evaluation, and model comparison.
  • Analyze model outputs for correctness, bias, safety, and hallucination using both qualitative and quantitative methods.
  • Build reusable test harnesses and datasets for benchmarking AI capabilities across tasks and domains.
  • Contribute to the development of internal tools for prompt engineering, evaluation visualization, and feedback loops.
  • Act as a quality advocate for AI-driven features across the product lifecycle.
  • Mentor junior team members in AI evaluation best practices and tooling.
  • Stay current with advancements in LLM evaluation, interpretability, and responsible AI.

Qualifications

  • Bachelor's or master's degree in computer science or a related technical field.
  • Minimum 6-8 years of experience into QA and Automation
  • Basic level understanding of software development lifecycle and quality best practices and tools
  • Good knowledge on Automation Framework Development
  • Proficient in the use of various automation frameworks like Playwright (preferred), Protractor, Rest Assured, karate etc.
  • Hands-on experience with OpenAI APIs (e.g., GPT-4, function calling, Assistants API).
  • Familiarity with OpenAI Evals and LangSmith for prompt orchestration and evaluation.
  • Strong Python skills, especially in building test automation and data analysis scripts.
  • Experience with prompt engineering, LLM fine-tuning, or RAG pipelines is a plus.
  • Proficiency in using tools like Jupyter, Pandas, and/or SQL for evaluation analytics.
  • Understanding of ethical AI principles, model alignment, and safety concerns.
  • Experience with version control (GitHub), CI/CD pipelines, and test management tools (e.g., JIRA, Azure Test Plans).
  • Excellent communication and collaboration skills, with the ability to explain complex AI behaviors to diverse stakeholders.

Bachelor's or master's degree in computer science or a related technical field.

  • QA Automation

    7 days ago

    Hyderabad, Telangana, India QA programmer Full time

    **ROLE :QA Automation (cypress)** **Location :Bangalore ,Hyderabad** **Experience:2+years** **Mandatory skills :Cypress** As a QA Automation Engineer specializing in Cypress, you will be responsible for designing, developing, and maintaining automated tests to ensure the high quality of our software products. You will work closely with cross-functional...

  • Engineering III Specialist-Software Development

    2 weeks ago

    Serilingampalli, Telangana, India Verizon Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    When you join VerizonYou want more out of a career. A place to share your ideas freely — even if they're daring or different. Where the true you can learn, grow, and thrive. At Verizon, we power and empower how people live, work and play by connecting them to what brings them joy. We do what we love — driving innovation, creativity, and impact in the...

  • Engineer III Spec-Data Engineering

    1 week ago

    Serilingampalli, Telangana, India Verizon Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    When you join VerizonYou want more out of a career. A place to share your ideas freely — even if they're daring or different. Where the true you can learn, grow, and thrive. At Verizon, we power and empower how people live, work and play by connecting them to what brings them joy. We do what we love — driving innovation, creativity, and impact in the...

  • Cloud Engineer Iii

    3 weeks ago

    Hyderabad, Telangana, India Jobted IN C2 Full time

    About McDonalds in India One of the world s largest employers with locations in more than 100 countries McDonald s Corporation has corporate opportunities in Hyderabad Our global offices serve as dynamic innovation and operations hubs designed to expand McDonald s global talent base and in-house expertise Our new office in Hyderabad will bring together...

  • QA Engineer

    2 weeks ago

    Hyderabad, Telangana, India NationsBenefits, LLC Full time

    Qualifications: - 7+ years in QA Automation, with 3+ years in a leadership role. - Experience with Software Development Life Cycle (SDLC), Software Testing Life Cycle (STLC), agile - model and different test methodologies & validation to ensure the Quality Assurance Control. - Experience using Relational Databases in Client/Server and Web environment -...

  • Sdet Iii

    3 days ago

    Kondapur, Hyderabad, Telangana, India Ivy Full time

    **Company Description** Ivy is a global, cutting-edge software and support services provider, partnering with one of the world’s biggest online gaming and entertainment groups. Founded in 2001, we’ve grown from a small tech company in Hyderabad to one creating innovative software solutions used by millions of consumers around the world, with billions of...

  • Sdet Iii

    21 hours ago

    Kondapur, Hyderabad, Telangana, India Entain plc Full time

    Company Description Ivy is a global, cutting-edge software and support services provider, partnering with one of the world's biggest online gaming and entertainment groups. Founded in 2001, we've grown from a small tech company in Hyderabad to one creating innovative software solutions used by millions of consumers around the world, with billions of...

  • QA Engineer

    2 weeks ago

    HITEC City, Hyderabad, Telangana, India Vertis Digital Sdn Bhd Full time

    **About Us** **Innovate. Create. Elevate.** At Vertis, we are on a mission to constantly innovate and create exceptional digital solutions. Our goal is to elevate our clients brands and the experiences they deliver to their customers through the power of technology-driven creativity. Our people are the catalyst, empowering our clients to successfully...

  • QA Automation Engineer

    2 weeks ago

    Hyderabad, Telangana, India karmanya Full time

    **Job Overview**: We are looking for a QA Automation Engineer to join our team of IT professionals. In this role, you will be suggesting ideas for automated software test procedures. Besides, you will be highlighting problem areas and tracking bugs that hinder software installation. You will also be designing and installing software databases. As a QA...

  • Software Engineer Iii

    3 weeks ago

    Hyderabad, Telangana, India JPMorgan Chase Full time

    Job Category Software Engineering We have an exciting and rewarding opportunity for you to take your software engineering career to the next level As a Software Engineer III at JPMorgan Chase within the AI ML Data Platform team you serve as a seasoned member of an agile team to design and deliver trusted market-leading technology products in a secure stable...