How Request Coalescing Stopped a Cache Stampede After Redis Failover
A Redis primary failure triggered a stampede that took the API down for 11 minutes. Request coalescing with a singleflight pattern cut thundering-herd errors to zero.
Caching · intermediate · 12 min read · 2298 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| API error rate during Redis failover | ~100% for 11 minutes | < 0.3% (retry window only) |
| Peak DB QPS during failover | 28,000+ | ~180 (coalesced) |
| P99 API latency during failover | 14,200 ms | 310 ms |
| Time to full traffic recovery | 11 minutes | 22 seconds |
| Synchronised TTL-expiry stampedes | 2 per week | 0 in 6 weeks post-deploy |
Key components
- singleflight.Group (Go) — Deduplicates concurrent cache-miss fetches within a process so at most one goroutine queries the database per unique key.
- Redis Sentinel — Monitors the Redis primary and promotes a replica automatically on failure, providing high availability for the cache tier.
- TTL jitter — Adds a random offset to each key's expiry time to prevent synchronised mass-expiry stampedes.
- Cache Warmer CronJob — Pre-populates the most-accessed keys before they expire, eliminating cold-start and post-failover stampedes.
- toxiproxy chaos experiment — Simulates Redis unavailability in staging to validate that the coalescing layer holds under realistic concurrent load.
- Prometheus singleflight metrics — Exposes deduplicated-request counts as an early warning signal for cache health degradation.
The Situation
This is a composite scenario assembled from patterns that recur across the industry; the organisation, names, and figures are illustrative and do not represent any single company.
The organisation in question is a 140-engineer B2B SaaS company that sells a workflow-automation platform to mid-market and enterprise customers. Their backend is a set of Go microservices deployed on AWS EKS, with a PostgreSQL 15 cluster on RDS as the system of record and a Redis 7 cluster (three-node, Sentinel-managed) as the primary caching layer. At peak, the platform handles roughly 28,000 requests per second across its API tier. The cache hit rate sat at a comfortable 94%, which meant the database was seeing only about 1,700 read queries per second under normal load — well within the read replica's capacity of around 12,000 QPS.
The team responsible for the incident was a six-person platform-reliability squad embedded within the broader infrastructure group. They owned Redis, the API gateway (Kong on EKS), and the internal Go SDK that all product services used to interact with the cache.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: caching, redis, reliability, golang, distributed-systems, incident-response