How Backoff, Jitter, and Circuit Breakers Stopped a Retry Storm from Killing a Production Database
A slow Postgres instance triggered aggressive client retries that amplified load 40x, causing a full outage. Exponential backoff, jitter, and a circuit breaker pattern restored stability within one sprint.
Distributed Systems · intermediate · 12 min read · 2325 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Outage duration from 41-second DB blip | 23 minutes | 0 minutes (graceful degradation) |
| Peak concurrent DB connection attempts during blip | ~24,000 | ~180 |
| P99 API latency during DB recovery | 28,000 ms | 620 ms |
| On-call pages per incident | 47 | 4 |
| SLA breach notifications triggered | 3 | 0 (across 3 subsequent maintenance windows) |
Key components
- Exponential Backoff with Full Jitter — Spreads retry attempts randomly across a time window, preventing synchronised thundering-herd load on the database during recovery.
- Circuit Breaker (pybreaker) — Opens after a failure threshold is crossed and returns fast errors to callers, stopping new connection attempts from reaching PgBouncer while the database recovers.
- Redis-backed Circuit Breaker State — Shares breaker state across all API and worker processes so that a breaker opened by one instance is immediately honoured by all others.
- PgBouncer in Transaction Mode — Reduces the number of server-side Postgres connections held open per client, creating headroom for burst traffic and reducing the blast radius of a connection storm.
- Shared Internal Library (platform-db-client) — Centralises retry and circuit-breaker logic so all 40 services inherit the fix from a single version bump without individual rewrites.
- Prometheus Circuit Breaker Metrics — Emits state-transition events that drive Alertmanager pages and Grafana annotations, making fast-fail behaviour visible during incidents.
The Situation
This is a composite scenario assembled from patterns that recur across the industry; the organisation, team structure, and figures are illustrative and do not represent any single company.
The organisation is a 120-engineer B2B SaaS company that sells a workflow-automation platform to mid-market and enterprise customers. The backend runs on AWS: roughly 40 Python (FastAPI) microservices, a handful of Node.js BFFs, all talking to a shared PostgreSQL 14 cluster on RDS (db.r6g.4xlarge primary, one read replica). Redis 7 on ElastiCache handles session data and short-lived caches. Celery workers process background jobs off SQS queues. Traffic peaks at roughly 12,000 requests per minute during business hours across North America and Europe.
The platform-reliability team is four engineers. They own the service mesh, observability stack (Prometheus, Grafana, Jaeger, Alertmanager), and deployment pipelines (GitHub Actions → ECS Fargate). They do not own individual services — that belongs to six product squads.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: distributed-systems, resilience, postgresql, circuit-breaker, retry-patterns, incident-response