Databricks AI Architect

The ideal candidate is a hands-on AI architect with deep Databricks and GenAI expertise, strong data engineering foundations, and a proven record of taking AI solutions to production at enterprise scale. Key requirements: 

• 12–14 years of experience in data and AI/ML solutions, including designing and architecting large-scale (TB/PB) data platforms and production AI/ML systems. 

• Deep, hands-on architecture expertise with Databricks — Spark/PySpark, Delta Lake, lakehouse/medallion architecture, Unity Catalog, Databricks Workflows, and Delta Live Tables — with strong performance and cost optimization. 

• Proven experience architecting Generative AI / LLM solutions — RAG pipelines, chunking and embedding strategies, vector store design, prompt engineering, evaluation, and guardrails/responsible-AI patterns. 

• Strong hands-on experience with native Databricks AI capabilities — Mosaic AI (Model Serving, Vector Search, Agent Framework), Foundation Model APIs, MLflow, Feature Store, and model governance via Unity Catalog. 

• Solid ML/MLOps foundations — model lifecycle management, experiment tracking, deployment, monitoring, and drift/quality evaluation in production. 

• Expert-level PySpark and Spark SQL, with strong Python for building reusable data and feature pipelines that power AI workloads (batch and streaming). 

• Strong hands-on experience with AWS and Azure Cloud, including networking, IAM/security, cost optimization, and AI/ML services; comfortable architecting across cloud data and AI services. 

• Experience with LLM orchestration frameworks (LangChain/LangGraph or equivalent), agentic patterns, tool use, and multi-agent workflows at an architecture level. 

• Deep expertise in data and AI governance, data quality, lineage, security, and responsible AI — including PII handling, access control (Unity Catalog, RBAC), and audit. 

• Proven ability to set architecture standards, lead design reviews, and provide technical leadership and mentorship across multiple engineering and data science teams. 

• Strong communication and stakeholder-management skills; able to drive architecture decisions and translate business needs into technical blueprints for both technical and executive audiences. 

Good to Have: Knowledge of LLMOps and observability tooling for AI systems; Databricks / cloud / AI certifications; experience leading cloud/data platform migration and modernization to Databricks; experience contributing to POCs, proposals, and RFPs.

AI & GenAI Architecture 

• Own the end-to-end architecture for AI/ML and GenAI solutions on Databricks — data and feature pipelines, model training/serving, RAG, and agentic applications. 

• Design RAG and LLM architectures — retrieval and embedding pipelines, vector store design, prompt/evaluation strategy, and guardrails — using Mosaic AI and Foundation Model APIs. 

• Define reference architectures, design patterns, and reusable frameworks for scalable, secure, and cost-optimized AI platforms. 

Data & Platform Foundation 

• Architect the lakehouse foundation (Delta Lake, medallion, Unity Catalog) and reusable PySpark pipelines that feed AI and analytics workloads. 

• Guide implementation using Databricks Workflows and Delta Live Tables, with standards for data quality, incremental loads, and schema evolution. 

MLOps, Governance & Reliability 

• Define MLOps/LLMOps practices — model lifecycle, MLflow tracking, deployment, monitoring, and drift/quality evaluation in production. 

• Establish AI and data governance, cataloging, lineage, access control, and responsible-AI standards (PII handling, guardrails, audit) ensuring security and compliance. 

• Define observability, monitoring, and cost-governance practices for reliable and efficient AI systems at scale. 

Technical Leadership & Collaboration 

• Set and enforce architecture standards and best practices; lead design and code reviews across multiple engineering and data science teams. 

• Mentor engineers, ML engineers, and leads, and provide hands-on guidance from prototype to production readiness. 

• Collaborate with product owners, engineering leaders, and business stakeholders to translate requirements into AI blueprints, POCs, and roadmaps.