Introduction to Model Monitoring Concepts
Know Exactly What to Watch in Production: Data Quality, Drift, Model Performance, and System Health, Each With a Threshold and an Owner.
By Sandip Das · · 5 min read
💡
Hey — It's Sandip Das 👋
In Day 40 we learned to detect drift with KS, PSI and Evidently. But drift is only one signal, and a drift report nobody looks at is just a nice HTML file.
Today we zoom out: what should you actually monitor in a production ML system, and how do you turn that into a written monitoring plan with thresholds, alerts and actions?
In Day 40 we learned to detect drift with KS, PSI and Evidently. But drift is only one signal, and a drift report nobody looks at is just a nice HTML file.
Today we zoom out: what should you actually monitor in a production ML system, and how do you turn that into a written monitoring plan with thresholds, alerts and actions?
📚 Key Learnings
- Why ML systems fail silently and why classic APM is not enough
- The four pillars: data quality, data drift, model performance, system health
- Online vs offline monitoring, and the ground-truth delay problem
- How to pick metrics, thresholds and alert severities that people will respect
- Hands-on: write a monitoring plan for iris-mlops as code (YAML) and validate it
🧠 Learn here
Why is ML monitoring different?
A normal web servic
Read the full article on learnxops.com.
Topics: 100 Days of MLOps, MLOps, ML, Monitoring