Building Performance Architect

Overview Own the end-to-end architecture of production AI systems with a strong hands-on bias. You’ll design robust, cost-efficient, and secure agentic/GenAI solutions on AWS. Part of your job will be to unblock lead developers by debugging code, optimizing performance, and guiding best practices. Define target architectures for agentic systems (planning/reasoning/tool-calling), GenAI/RAG pipelines, and evaluation loops; produce clear design documents with Flow/UML/sequence diagrams and AWS deployment topologies. • Size and optimize infrastructure for cost and performance: model throughput/latency, concurrency, autoscaling policies, CPU/GPU needs, memory footprints, vector index sizing, storage/egress, and token budgets. • Establish reference implementations for multi-agent frameworks (Semantic Kernel preferred; LangGraph/AutoGen/CrewAI acceptable), tool/function schemas, validation, memory, grounding, and multi-step planning. • Architect retrieval and hybrid search systems: ingestion, chunking, embeddings, ranking, caching, freshness, and grounding; evaluate recall, precision, and hallucination risk. • Productionize on AWS using Amazon EKS, S3, SQS/SNS, and AWS Bedrock; integrate identity (Okta/IAM), secrets (AWS Secrets Manager), eventing, and observability; Make systems observable: distributed tracing, metrics, and logs using OpenTelemetry and Datadog; Build evaluation and promotion workflows: prompt/flow tests, golden sets, offline batch runs, A/B experiments, regression suites, and rollout gates. • Design security and safety controls: threat modeling, prompt-injection defense, sandboxed tools, policy enforcement, red-team testing, PII/data governance, and audit trails. • Define platform standards: reusable SDKs, connectors, CI/CD templates, runbooks, and architecture review checklists. • Partner with product, data, and SRE teams to plan capacity, disaster recovery, multi-region posture, and upgrade paths; 7–10 years in software/AI engineering with at least 4+ years building GenAI applications and 2+ years architecting production agentic systems. • Strong hands-on expertise in Python 3.11+ (typing, asyncio, packaging, profiling, pytest); able to dive into code, fix bugs, and optimize performance-critical paths. • Experience with one or more agent frameworks (Semantic Kernel, LangGraph, AutoGen, CrewAI) and function/tool calling with schema and argument validation. • Proven design of GenAI/RAG/hybrid retrieval systems using AWS Bedrock, OpenSearch Serverless, or other vector databases; grounding and retrieval evaluation experience. • Deep knowledge of AWS architecture: Amazon EKS, Bedrock, S3, SQS/SNS, RDS (SQL Server/PostgreSQL), ElastiCache (Redis), Secrets Manager, IAM/Okta, Kong API Gateway, OpenSearch Serverless, and Datadog. • mature incident response and on-call practices. • Cost and performance engineering mindset: capacity modeling, GPU/CPU sizing, autoscaling (HPA), batching/streaming, caching, and FinOps discipline. • Security and safety fundamentals: least privilege, data isolation, policy enforcement, content moderation, jailbreak/PII defenses, and compliance awareness. • Excellent technical communication: clear diagrams, ADRs, design docs; empathetic, structured code and architecture reviews. Multi-agent orchestration patterns: task decomposition, coordinator-worker, human-in-the-loop, graph-based planning. • Deep expertise with vector databases and retrieval: Evaluation frameworks: red teaming, automated guardrails, regression testing, rollout gates, canary deployments. • Frontend integration for agent UIs (streaming responses, tool traces), secure connector APIs, and AuthN/Z best practices. • Policy-as-code (OPA) and multi-tenant architecture (RBAC, quotas, usage metering). • CI/CD exposure (build/test with GitHub Actions, deployments via Terraform/AWS IaC templates). Core Tech Stack (our core; Python 3.11+, FastAPI, Pydantic v2, SQLAlchemy 2.Amazon EKS, AWS Bedrock, Amazon SQS/SNS, Amazon RDS (SQL Server/PostgreSQL), ElastiCache (Redis). • AWS Secrets Manager, Okta IAM, NeMo Guardrails, Kong API Gateway. • OpenTelemetry + Datadog for observability and monitoring. • Custom RAG Services, Bedrock Knowledge Base, and LLM evaluation with Phoenix, Arize, and Promptfoo.