Custom ML Metrics with Prometheus
Go Beyond Latency: Export Predictions per Class, Confidence Scores, Feature Statistics, and Model Version So Your Dashboard Shows What the Model Is Doing.
By Sandip Das · · 4 min read
💡
Hey — It's Sandip Das 👋
In Day 43 we scraped the FastAPI app with Prometheus and built request rate, error and latency panels. Those tell us the server is healthy, but a perfectly healthy server can still be serving a confused model.
Today we add model-level metrics with the Prometheus Python client: predictions per class, confidence scores, input feature statistics and the model version that is live.
In Day 43 we scraped the FastAPI app with Prometheus and built request rate, error and latency panels. Those tell us the server is healthy, but a perfectly healthy server can still be serving a confused model.
Today we add model-level metrics with the Prometheus Python client: predictions per class, confidence scores, input feature statistics and the model version that is live.
📚 Key Learnings
- The four Prometheus client types in Python: Counter, Gauge, Histogram, Summary
- Which ML signals to export: class distribution, confidence, feature stats, model version
- Label design and cardinality: what is safe to put in a label
- Choosing histogram buckets for scores between 0 and 1
- Hands-on: custom metrics in the iris FastAPI app plus PromQL to chart them
🧠 Learn here
Why c
Read the full article on learnxops.com.
Topics: 100 Days of MLOps, MLOps, ML, Monitoring