How Anomaly Detection Cut Alert Noise by 73% and Caught Incidents 18 Minutes Earlier
A B2B SaaS platform drowning in threshold alerts rebuilt its observability pipeline around ML-based anomaly detection, slashing noise and catching real incidents before customers noticed.
Observability & SRE · intermediate · 12 min read · 2308 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Weekly alert volume | 340 alerts/week | 92 alerts/week |
| False positive rate | 77% | 21% |
| Mean time to detect (MTTD) | 34 minutes | 16 minutes |
| On-call pages per engineer per week | 8.5 | 2.3 |
| Customer-reported incidents (rolling 90 days) | 3 in prior 6 months | 0 in 90 days post-rollout |
Key components
- Isolation Forest (scikit-learn) — Multivariate anomaly scoring model that assigns a 0–1 anomaly score to each incoming feature vector without requiring labelled training data.
- Kafka (AWS MSK) — Central streaming bus that decouples metric ingestion from scoring, allowing the detection layer to be swapped or scaled independently.
- prometheus-kafka-adapter — Bridges Prometheus remote-write to Kafka so existing scrape configs require only a single additional remote_write target.
- Alertmanager webhook receiver — Gates alert delivery on the scorer's confidence score, routing low-confidence events to a suppressed log rather than paging on-call engineers.
- MLflow Model Registry (S3-backed) — Stores versioned, serialised models so the scorer can hot-reload after weekly retraining without a deployment.
- Evidently AI drift detection — Monitors Jensen-Shannon divergence between live and training feature distributions and triggers automated retraining when drift exceeds the configured threshold.
- SHAP explainability layer — Computes per-feature Shapley values for each anomaly score and surfaces them in Grafana so on-call engineers understand why an alert fired.
The Situation
This is a composite scenario assembled from patterns observed across multiple production observability projects; all organisations, team sizes, and figures are illustrative and internally consistent.
The organisation is a 140-engineer B2B SaaS company running a multi-tenant workflow-automation platform. Engineering is split into eight product squads plus a four-person SRE team that owns the observability stack. The platform runs on AWS: roughly 200 ECS Fargate tasks, 14 RDS PostgreSQL clusters, two MSK (Managed Kafka) clusters, and a handful of Lambda functions for async jobs. Metrics are scraped by Prometheus (self-hosted on EC2), traces flow through an OpenTelemetry Collector into Tempo, and logs land in Loki — all visualised in Grafana. PagerDuty handles on-call routing. Monthly active tenants sit at around 3,400, and the SLA commits to 99.9% uptime.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: observability, anomaly-detection, alerting, prometheus, mlops, sre