The 73-Hour Roblox Outage: How Consul Streaming and BoltDB Contention Took Down 50 Million Users
In October 2021, a Consul misconfiguration triggered BoltDB write contention that cascaded into a 73-hour total outage for Roblox, affecting 50M+ daily users. The lesson: service-mesh health depends on the databases underneath it.
Distributed Systems · advanced · 12 min read · 2131 words · By Sandip Das · Published
Public postmortem: based on what the organisation published. Sources are listed below.
What this covers
- Background
- What Happened
- Root Cause
- Detection and Response
- Impact
- The Fix
- Lessons for Your Team
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Outage duration | Platform fully operational | ~73 hours of total service unavailability (28–31 Oct 2021) |
| Users affected | 50M+ daily active users served | All 50M+ daily active users unable to access the platform |
| Services affected | All services healthy | All user-facing services down: gameplay, website, mobile app, developer portal |
Key components
- HashiCorp Consul — Service-discovery and health-check backbone; its degradation made the entire platform unable to route traffic.
- BoltDB (embedded in Consul) — Single-writer embedded key-value store backing Consul's Raft log; became the write-contention bottleneck under high streaming load.
- Consul Streaming — Push-based catalog update mechanism that replaced polling; dramatically increased write load on BoltDB when enabled at Roblox's subscriber scale.
- Raft consensus log — Distributed log used by Consul servers to maintain catalog consistency; its commit latency was the leading indicator of BoltDB saturation.
- Thundering herd (reconnect storm) — Pattern where all Consul agents reconnected simultaneously after each server restart, re-saturating the cluster and extending the outage from hours to days.
Background
Roblox is an online gaming platform that, by late 2021, hosted over 50 million daily active users and hundreds of thousands of concurrent game servers. The platform is not a single game but an infrastructure layer on which millions of user-created experiences run. Keeping those experiences reachable requires a service-discovery and health-check mesh that operates at enormous scale: thousands of backend services register themselves, deregister, and query each other continuously.
Roblox runs HashiCorp Consul as its service-discovery backbone. Consul agents run on every host; a small cluster of Consul servers maintains the authoritative catalog of which services are healthy and where they live. Every game server, API service, and internal component depends on Consul to find its dependencies. When Consul is degraded, the entire platform loses the ability to route traffic correctly.
The October 2021 outage lasted approximately 73 hours — from roughly 16:30 PDT on 28 October to around 17:00 PDT on 31 October — and was one of the longest total outages of a major consumer internet platform in recent memory. Roblox published a detailed Return to Service postmortem in January 2022.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Sources
Topics: distributed-systems, consul, service-discovery, incident-response, infrastructure, postmortem