MLOps Observability Engineer

The MLOps Observability Engineer will design, implement, and maintain the comprehensive monitoring, logging, and tracing solutions for our entire ML platform and production models. This includes building automated systems to detect model decay, data drift, and infrastructure performance issues, ensuring that our AI/ML applications are reliable, scalable, and maintain continuous business value.

II. Key Responsibilities :

A. MLOps Monitoring and Model Health Design and Implement Model Observability : Define and implement Model Performance Monitoring metrics (e.g., accuracy, precision, recall, RMSE, AUC) and business-impact metrics for deployed ML models.

- Data and Concept Drift Detection : Build and automate data quality and validation checks to continuously monitor for data drift, concept drift, and data integrity issues that could degrade model performance.

- Explainability and Fairness Monitoring : Implement tools and techniques for model interpretability and model explainability (XAI), tracking feature importance, and monitoring for potential bias or fairness issues in production.

- Alerting and Triage : Establish clear, actionable alerting thresholds for model and infrastructure degradation, integrating with incident management workflows for quick triage and resolution.

B. Observability Platform and Infrastructure Telemetry Pipeline Development : Design, deploy, and manage robust Observability pipelines to collect, aggregate, and route the three pillars of observability (metrics, logs, and traces) from the ML platform and inference services.

- Dashboarding and Visualization : Create insightful and real-time dashboards (SLIs/SLOs) to provide a clear, unified view of the ML system's health, from infrastructure load to model prediction quality.

- Infrastructure-as-Code (IaC) for Observability : Use IaC tools to provision and manage the monitoring and logging infrastructure across cloud environments.

- Cost Optimization : Monitor telemetry data costs, implementing smart sampling and retention policies to ensure efficient use of observability tools.

C. Automation and CI/CD Automated Retraining Triggers : Integrate observability signals (like performance drop or data drift alerts) to automatically trigger the ML pipeline (CI/CD) for model retraining, testing, and redeployment.

- Reproducibility and Auditing : Ensure that model monitoring and all MLOps processes are fully reproducible, traceable, and adhere to governance and regulatory standards.

- Collaboration and Consultation : Work closely with Data Scientists and ML Engineers to instrument new models for observability from the ground up, educating them on best practices for monitoring and logging.

III. Technical Skills and Qualifications

A. Programming and Scripting Expert : Proficiency in Python is required, including libraries for data manipulation and ML (e.g., NumPy, Pandas, Scikit-learn).

- Strong Shell Scripting (Bash/Zsh) and experience with other languages like Go or Java is a plus.

B. MLOps Tools and Frameworks ML Frameworks : Familiarity with TensorFlow, PyTorch, or Scikit-learn to understand how models are built and served.

- MLOps Platforms/Tools : Hands-on experience with MLflow, Kubeflow, Data Version Control (DVC), or comparable solutions for experiment tracking and model registry.

- Orchestration : Experience with pipeline orchestration tools like Airflow, Kubeflow Pipelines, or Argo Workflows.

C. Cloud and Containerization Cloud Platforms : Deep working knowledge of one or more major cloud providers (AWS, GCP, or Azure) and their ML services (e.g., AWS SageMaker, Google AI Platform, Azure ML).

Containerization & Orchestration :

- Expertise with Docker and Kubernetes for deploying and managing production ML services.

- Infrastructure as Code (IaC) : Proficiency with Terraform or Ansible for infrastructure automation.

D. Monitoring and Observability Stack Metrics & Time-Series :

- Expertise with Prometheus and Grafana for collecting, querying, and visualizing time-series data.

- Logging & Tracing : Experience with centralized logging solutions (ELK Stack/Elasticsearch, Loki, Splunk) and distributed tracing tools (Jaeger, Zipkin, OpenTelemetry).

- Model Monitoring Tools : Experience with specialized model performance monitoring tools like Evidently AI, Seldon Core, or similar internal/commercial tools.

IV. Education and Experience :

Education : Bachelors or Masters degree in Computer Science, Software Engineering, Data Science, or a related technical field.

Experience : 3 years of experience in an MLOps, DevOps, SRE, or Observability-focused engineering role, with at least 1-2 years dedicated to production ML systems.

Soft Skills : Excellent problem-solving, analytical skills, and strong communication for collaborating effectively with cross-functional teams (Data Science, Software Engineering, Product).