How a Five-Person SRE Team Cut Operational Toil by 50% with Automation and Observability
A small SRE team drowning in manual alerts and runbooks automated their way to half the toil in 90 days using Terraform, PagerDuty, and a self-healing Kubernetes operator.
Fundamentals · beginner · 12 min read · 2401 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Weekly toil hours per engineer | 22 h | 11 h |
| False-positive page rate | 55% | 18% |
| MTTR for P2 incidents | 47 min | 19 min |
| P2/P3 incidents resolved without human action | 0% | 61% |
| SLA breaches (rolling quarter) | 2 | 0 |
Key components
- PagerDuty Event Orchestration — Classifies every incoming alert as P1, P2, or P3 and routes it to the correct automation path or human responder.
- Self-Healing Kubernetes Operator — Watches for P2 PagerDuty webhooks and executes pre-approved Kubernetes API remediations automatically, with cooldown and audit logging.
- Runbook Bot — Executes Git-backed, approved runbook scripts as Kubernetes Jobs in response to P3 alerts, posting results to Slack.
- Prometheus Alert Rule Audit — Reduced active alert rules from 340 to 180 and added inhibition rules, cutting false-positive page rate by more than half.
- Immutable S3 Audit Log — Satisfies SOC 2 requirements by recording every automated action with object-lock enabled, without slowing down the automation.
- Terraform Runner via CodeBuild — Allows on-call engineers to trigger infrastructure changes via a Slack slash command backed by IRSA-scoped IAM, eliminating manual plan-and-apply cycles.
The Situation
This is a composite scenario built from patterns observed across multiple SRE teams in the industry; the organisation, names, and figures are illustrative and do not represent any single real company.
The organisation is a 140-engineer B2B SaaS company selling a workflow-automation product to mid-market finance and logistics customers. The platform runs on AWS, with roughly 60 microservices deployed on EKS across two regions (us-east-1 and eu-west-1). The SRE team is five people: one senior, three mid-level, and one junior. They own reliability, on-call, deployment pipelines, and infrastructure-as-code. The platform serves about 4,000 active tenants, processes 12 million API calls per day, and has a contractual 99.9% monthly uptime SLA.
The stack at the start of this story: EKS 1.27, Terraform 1.5 for infrastructure, GitHub Actions for CI/CD, Prometheus and Grafana for metrics, Loki for log aggregation, PagerDuty for alerting, and a Confluence wiki full of runbooks that nobody kept up to date.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: sre, automation, kubernetes, observability, toil-reduction, on-call