How a B2B SaaS Team Cut Weekly Alerts from 400 to 18 with SLO-Based Alerting
A 120-engineer SaaS company replaced threshold spam with error-budget burn-rate alerts, dropping weekly pages from 400 to 18 and cutting MTTR by 60%.
Observability & SRE · intermediate · 12 min read · 2337 words · By Sandip Das · Published
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Weekly alert notifications | ~400 | ~18 |
| Actionable alert ratio | ~3% | ~89% |
| Mean time to acknowledge (MTTA) | 18 min | 6 min |
| Mean time to resolve (MTTR) | 52 min | 21 min |
| On-call fatigue score (10 = worst) | 7.4 | 3.1 |
Key components
- Sloth SLO Engine — Generates Prometheus recording rules and multi-window burn-rate alert rules from a simple YAML SLO spec.
- Multi-window burn-rate alerting — Fires only when the error rate is high enough and sustained long enough to meaningfully consume the error budget, eliminating flapping threshold alerts.
- Prometheus Ruler — Evaluates Sloth-generated recording rules on a schedule, pre-computing error-ratio windows so alert expressions stay cheap.
- Alertmanager routing split — Routes fast-burn (critical) alerts to PagerDuty and slow-burn (warning) alerts to Slack, matching urgency to response channel.
- Error budget report script — Nightly Python script that queries Prometheus and posts remaining error budget per SLO to Slack, making reliability visible to the whole team.
- Parallel run period — Six-week overlap of old threshold alerts and new SLO alerts, used to validate coverage before decommissioning legacy rules.
The Situation
The organisation is a 120-engineer B2B SaaS company selling workflow-automation software to mid-market and enterprise customers. The platform processes roughly 4 million API requests per day across four core services: an API gateway, an order-processing service, a payment service, and an async notification service. The backend runs on Kubernetes (EKS, 14 nodes across three availability zones), with PostgreSQL on RDS, Redis on ElastiCache, and Kafka on MSK. The observability stack at the start of this story was Prometheus + Alertmanager for metrics, with 230 hand-crafted alert rules accumulated over three years, and Grafana for dashboards.
The SRE team was four people. On-call rotated across eight engineers from the broader backend group.
---
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: observability, slo, alerting, prometheus, grafana, sre