How a Retail Platform Survived 10x Black Friday Traffic With Load Testing and Game Days
A mid-size e-commerce platform used k6, chaos engineering, and capacity planning sprints to handle a 10x traffic spike on Black Friday without a single P1 incident.
Fundamentals · intermediate · 12 min read · 2383 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Peak rps without errors | ~600 rps | 2,600 rps |
| p99 checkout latency at peak | 4,200 ms | 620 ms |
| Error rate at peak | 18% | 0.3% |
| DB failover recovery time | 47 minutes | 4 minutes |
| Black Friday P1 incidents | N/A (no prior test) | 0 |
Key components
- k6 load testing — Drives synthetic traffic at defined rps targets using realistic user journey scripts with constant-arrival-rate executors.
- Kong Gateway rate limiting — Enforces per-consumer and per-IP rate limits so that burst traffic sheds gracefully with HTTP 429 rather than cascading into backend services.
- pgBouncer (transaction mode) — Pools Postgres connections in front of RDS to prevent connection exhaustion under high concurrency.
- Kafka event bus — Decouples email notification and inventory sync from the synchronous checkout path, removing latency and failure dependencies.
- Game days — Structured failure injection exercises that test the team's incident response and runbooks, not just the system's technical resilience.
- Prometheus and Grafana — Provide real-time metrics and SLO dashboards during load tests and game days, enabling fast identification of bottlenecks.
- LaunchDarkly feature flags — Allow non-critical features like personalisation and review ingestion to be disabled instantly during peak traffic to reduce load on Elasticsearch.
The Situation
This is a composite scenario assembled from patterns observed across multiple e-commerce and retail SaaS organisations; all figures are illustrative and chosen for internal consistency, not to represent any single company.
The organisation is a 90-engineer direct-to-consumer retail platform that processes roughly 40,000 orders per day during normal trading. The stack is a microservices architecture running on AWS EKS (Kubernetes 1.28), with Kong as the API gateway, Next.js on the frontend, and a mix of Node.js and Go services on the backend. Persistence is split across RDS Postgres (multi-AZ), a Redis 7 cluster for sessions and catalogue caching, and Elasticsearch for product search. Events flow through a self-managed Kafka cluster on EC2. Observability is handled by Prometheus, Grafana, and Loki, all deployed in-cluster.
The platform had never seen a traffic event larger than 3x its daily baseline. Black Friday was eight weeks away, and the growth team's projections — based on a 60% year-on-year uplift in the customer base plus a planned TV campaign — pointed to a 10x peak: roughly 2,400 requests per second at the API gateway versus the usual 240 rps.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: load-testing, capacity-planning, chaos-engineering, kubernetes, observability, reliability