Building an Incident Response Program from Scratch: On-Call, Severities, and Blameless Reviews
A 120-engineer B2B SaaS company had no formal incident process, causing multi-hour outages with no coordination. They built on-call rotations, severity tiers, and blameless postmortems—cutting MTTR from 4.2 hours to 38 minutes.
Fundamentals · beginner · 12 min read · 2244 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Mean Time to Respond (MTTR) | 4.2 hours | 38 minutes |
| Mean Time to Detect (MTTD) | 28 minutes | 4 minutes |
| Customer-visible incidents per month | 3–4 | 0–1 |
| Alert noise (pages per week) | 47 | 9 |
| Engineer on-call satisfaction (1–5) | 2.1 | 3.8 |
Key components
- PagerDuty Escalation Policy — Routes pages from Prometheus, Loki, and Sentry to the correct on-call engineer with timed escalation to secondary and manager.
- Severity Matrix — Four-level classification (SEV1–SEV4) with concrete examples that determines response actions, war-room creation, and Statuspage updates.
- Slack War Room Automation — Auto-creates a dedicated incident channel on PagerDuty webhook, providing a single coordination thread with timestamped updates.
- Runbook Store (GitHub) — Markdown runbooks linked directly from PagerDuty alerts, covering immediate mitigation, root cause investigation, and escalation paths.
- Blameless Postmortem Process — 48-hour postmortem with a dedicated facilitator, Notion template, and Jira action items reviewed weekly to close the improvement loop.
- Argo CD Rollback Runbook — One-command rollback procedure that reduces resolution time for bad-deploy incidents from hours to minutes.
The Situation
This is a composite scenario built from patterns common across the industry; the organisation, team structure, and numbers are illustrative and do not represent any single real company.
The organisation is a 120-engineer B2B SaaS company selling workflow-automation software to mid-market finance and HR teams. Engineering is split into eight product squads plus a four-person platform team. The stack is a Kubernetes-on-EKS monorepo broken into roughly 30 microservices, a primary PostgreSQL 15 cluster on RDS, Redis for session and job-queue caching, and an Nginx-based API gateway. Deployments run through Argo CD; observability is Prometheus and Grafana for metrics, Loki for logs, and Sentry for application errors. Monthly active users: around 18,000. Annual recurring revenue: in the $14 M range. SLA commitments to enterprise customers: 99.9% monthly uptime.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: incident-response, on-call, sre, postmortem, observability, fundamentals