AI Quality

Job Description The AI Quality & Safety Engineering Lead ensures that AI systems across the enterprise are safe, fair, reliable, compliant, and observable from prerelease through continuous operations. This role designs and operates enterprise evaluation frameworks (quality, bias/fairness, robustness, privacy), leads redteaming and guardrail development, encodes policy and regulatory requirements as automated checks, and drives metric reporting and postincident learning across AI programs. The lead partners closely with Enterprise Architecture, Compliance, Legal/Privacy, Security, Data Governance, and product/engineering teams to enforce quality and safety gates and enable responsible scale. (Aligned to Corporate Policy and Responsible AI principles.) Reporting into the Head of Applied AI Engineering, you will manage a team of QA Engineers and AI Engineers. This role blends deep technical leadership, product engineering strategy, and direct enablement of enterprise use cases and services. You will partner closely with business, legal and technical stakeholders to design and deliver robust, scalable solutions that demonstrate the value of AI across our enterprise in a safe and compliant way. This role is execution-oriented with strong influence on building AI solutions responsibly. Evaluation and Test Strategy - Define and run evaluation frameworks for LLMs and multi-modal systems covering performance, fairness, bias, robustness, safety, and privacy. - Build automatic + human-in-the-loop evaluation pipelines with reproducible scoring, statistical significance, and error analysis. - Establish SLOs/SLIs for quality (e.g., grounded accuracy, hallucination rate, privacy leak rate, harmful-content rate). Red-Teaming & Safety Guardrails - Design and run red-teaming against prompt injection, jailbreaks, sensitive data leakage, tool-abuse, and retrieval abuse. - Develop guardrail solutions (classification, content safety, PII/PHI detection, grounding checks, allow/deny lists, rate limiting, output filtering) to mitigate hallucinations, leakage, and compliance violations. - Maintain attack libraries and scenario banks; simulate adversarial behavior and monitor adaptive attacks over time. PreRelease Gating & Continuous Assurance - Operate prerelease evaluation gates with baselines, regression checks, canary/AB testing, and rollback criteria. - Run continuous evaluation in production with synthetic & real traffic (shadow/ghost testing) and drift, bias, and toxicity monitoring. PolicyasCode & Compliance - Partner with Enterprise Architects, AI Platforms and Compliance/Legal to encode policies as automated checks in CI/CD (policy-as-code) and runtime enforcement. - Align with frameworks such as NIST AI RMF, ISO/IEC 23894 & 42001, GDPR, and the EU AI Act risk management obligations. - Produce model/system cards and traceability artifacts for audits. Stakeholder Leadership & Culture - Coach product teams on responsible AI practices and establish quality ownership norms. - Build a community of practice across Engineering, Security, Data Governance, and Riskstandardizing test data, benchmarks, and review rituals. - Mentor engineers building evaluation, red-team, and guardrail capabilities. Basic Qualifications - 7+ years in ML/AI/Platform/Quality roles, including 3+ years evaluating production systems. - Advanced degree in Computer Science, Engineering or related discipline. - Proven design & operation of evaluation harnesses and policy-as-code checks in CI/CD. - Hands-on Python and expertise with evaluation and safety tooling such as: OpenAI Evals, Ragas/LangSmith, DeepEval, Azure AI Content Safety / OpenAI Moderation, Presidio, Guardrails.ai, Nemo, MLFlow, Grafana, Kibana, Azure monitor - Strong grasp of red-team techniques (prompt/indirect injection, jailbreaks, training data extraction, membership inference, model poisoning, tool abuse). - Cloud & MLOps: Azure/AWS, CI/CD (GitHub Actions/Azure DevOps, feature stores, vector DBs (Azure AI Search, OpenSearch, Pinecone), testing in pipelines. - Regulatory literacy (e.g., GDPR, EU AI Act readiness), and experience collaborating with Compliance/Risk, Security, and Legal. - Excellent communication with executives and engineers - Experience leading an engineering team. Preferred Qualifications - Led an AI safety/quality function or enterprise red-team program. - Prior work with AI based applications, tool use/orchestration, and enterprise data governance. - Experience with retrieval grounding checks, memory governance, and multi-tenant safety controls. Please apply by sending your CV and a motivational letter in English Purpose Breakthroughs that change patients lives... At Pfizer we are a patient centric company, guided by our four values: courage, joy, equity and excellence. Our breakthrough culture lends itself to our dedication to transforming millions of lives. Digital Transformation Strategy One bold way we are achieving our purpose is through our company wide digital transformation strategy. We are leading the way in adopting new data, modelling and automated solutions to further digitize and accelerate drug discovery and development with the aim of enhancing health outcomes and the patient experience. Flexibility We aim to create a trusting, flexible workplace culture which encourages employees to achieve work life harmony, attracts talent and enables everyone to be their best working self. Let's start the conversation Equal Employment Opportunity We believe that a diverse and inclusive workforce is crucial to building a successful business. As an employer, Pfizer iscommitted to celebratingthis,in all itsforms allowing for us to be as diverse as the patients and communities we serve. Together, we continue to build a culture that encourages, supports and empowers our employees. Disability Inclusion Our mission is unleashing the power of all our people and we are proud to be a disability inclusive employer, ensuring equal employment opportunities for all candidates. We encourage you to put your best self forward with the knowledge and trust that we will make any reasonable adjustments to support your application and future career. Your journey with Pfizer starts here Information & Business Tech