AI Model Assurance Specialist

2 days ago

solapur, India beBeeAssurance Full time

Job Title: AI Model Assurance SpecialistDescription:The AIRI Quality Framework emphasizes the importance of ensuring advanced Large Language Models (LLMs) and multi-agent systems meet performance, reliability, and trustworthiness benchmarks. As an AI Model Assurance Specialist, you will play a pivotal role in guaranteeing these models operate within specified parameters.Key Responsibilities:Evaluation Strategy Design: Collaborate with Data Science teams to develop and execute comprehensive evaluation plans for LLMs, sub-models, and multi-agent systems.Testing and Automation: Perform manual and automated testing of AI models, ensuring thorough coverage of all critical scenarios.Synthetic Data Generation: Utilize Python scripting to create realistic test data simulating real-world scenarios and edge cases, thereby enhancing model robustness.Boundary Condition Coverage: Ensure adequate automated data generation techniques are in place to cover all potential boundary conditions.Prompt Engineering: Develop and maintain high-quality prompt libraries for model evaluation across diverse use cases.Multi-Model Architecture Evaluation: Assess multi-model and multi-agent architectures to ensure they meet functional and performance standards.Evaluation Metrics Interpretation: Apply and interpret relevant evaluation metrics, demonstrating a deep understanding of their formulas and practical implications.Team Collaboration: Work closely with Data Science and Engineering teams to align QA strategies with model development, fostering an environment of collaboration and open communication.Documentation: Maintain accurate and detailed documentation of test plans, test cases, and evaluation reports for stakeholder review and signoff.Requirements:5+ Years Experience: Minimum 5 years of relevant experience in quality assurance and AI/ML evaluation.LLM Evaluation Techniques: Strong hands-on experience with Large Language Model evaluation techniques.Python Proficiency: Proficient in Python programming for scripting, test automation, and synthetic data generation.AI Model Architecture: Deep understanding of AI model architecture, including models, sub-models, and agent-based systems.Prompt Engineering: Familiarity with prompt engineering principles and its role in model testing.AI/ML Testing Frameworks: Experience with AI/ML testing frameworks and tools.Evaluation Metrics: Solid grasp of evaluation metrics and their practical application.Analytical Skills: Excellent analytical, documentation, and communication skills.Prior QA Experience: Prior experience in QA for AI/ML products.Tool Proficiency: Familiarity with tools like RAGAS, DeepEval, Arize Phoenix, OpenAI APIs, or similar.CI/CD Pipelines: Experience with CI/CD pipelines for AI model deployment and testing.Benefits:Collaborative Environment: Opportunity to work in a dynamic, collaborative environment that fosters innovation and knowledge sharing.Professional Growth: Potential for professional growth and development through continuous learning and skill enhancement.Competitive Compensation: Competitive compensation package that reflects individual contributions and value added to the organization.Note:Sense of Humor: A sense of humor is appreciated but not required.

  • Quality Assurance Specialist

    1 week ago

    solapur, India beBeeValidation Full time

    Job OpportunityWe are seeking a Lead Quality Assurance Specialist – AI Validation with strong expertise in Python automation, Large Language Model evaluation, Risk and Alert Generation pipelines, observability, adversarial testing, and Azure monitoring.Main Responsibilities:Leading quality assurance initiatives for cutting-edge Generative AI and...

  • Model Development Trainee – AI

    1 week ago

    Solapur, India NetPrompt AI Full time

    Model Development Trainee – AI & LLMs (0–2 yrs)NetPrompt AI | 100% Remote | Flexible Hours |⏳ 6 Months | NVIDIA University Elite AccessCompany: NetPrompt AI (Early-Stage GenAI Startup)Program Type: High-Impact AI Model Development Trainee ProgramPowered By: NVIDIA Inception & Exclusive NVIDIA University Training CreditsThe Opportunity: Build the...

  • AI Specialist

    2 weeks ago

    solapur, India beBeeArtificial Full time

    As an AI specialist, you will be responsible for designing and deploying intelligent systems.Your key responsibilities include developing and maintaining AI models, integrating them with cloud-based services, and implementing scalable automation workflows.You will work on architecting robust pipelines, developing secure chatbots, and integrating vector...

  • Data Assurance Specialist

    1 week ago

    solapur, India beBeeDataAssurance Full time

    Job OpportunityWe are seeking a qualified Data Assurance Specialist to fill this key role.Develop and execute comprehensive test strategies for data ingestion, dbt transformation models, and system functionality.Create automated and manual tests to ensure high-quality data, completeness, and accuracy across Snowflake environments.Validate model integrity and...

  • AI Specialist

    1 week ago

    solapur, India beBeeMachine Full time

    Job Title: AI SpecialistWe are seeking an experienced AI Specialist to join our team of experts in machine learning engineering. The ideal candidate will have at least 10+ years of experience in designing, building, and deploying ML models into production-grade environments.The successful candidate will be responsible for setting up and managing MLOps...

  • AI Model Specialist

    2 weeks ago

    solapur, India beBeeModel Full time

    Artificial Intelligence Model BuilderThe role of an Artificial Intelligence Model Builder involves designing and developing complex statistical models using various programming languages. With 12-18 years of experience in data science and machine learning, this individual will be responsible for creating models that can solve real-world problems.This...

  • AI Model Developer

    1 week ago

    solapur, India beBeeDeveloper Full time

    AI Model DeveloperAs a highly skilled AI model developer, you will be responsible for designing, developing, and deploying AI models that power intelligent chatbots, voice agents, and telephony services. This is an excellent opportunity to leverage your expertise in machine learning and artificial intelligence to drive business growth.Key...

  • AI Model Designer

    2 days ago

    solapur, India beBeeSports Full time

    Machine Learning DeveloperThis role requires expertise in designing and developing AI models for Sports Companion GPT - Aiko.The ideal candidate will combine conventional ML techniques with generative AI models to create insights, predictions, or content tailored to Cricket and Football applications.A robust evaluation framework must be developed and...

  • AI Quality Assurance Professional

    1 week ago

    solapur, India beBeeArtificialIntelligence Full time

    Senior Test EngineerWe are seeking a highly skilled and experienced AI Quality Assurance professional to join our dynamic team.Key Responsibilities:Required Skills and Qualifications:Benefits of Working with Us:About the Role:This is a senior role that requires extensive experience in testing AI and Machine Learning products. The ideal candidate will have a...

  • Ai software engineer

    3 weeks ago

    Solapur, India Taskify AI Full time

    AI Software Engineer Summary We are seeking a talented AI Software Engineer to join our engineering team and drive the design, development, and deployment of AI-powered solutions. This role is ideal for professionals passionate about artificial intelligence, machine learning, and software engineering who want to make a tangible impact on real-world...