The Fastly June 2021 Outage: How One Customer Config Triggered a Global CDN Collapse
A single customer's valid configuration change activated a latent bug in Fastly's VCL software, taking 85% of the network offline for 49 minutes and knocking out major internet properties worldwide.
Caching · intermediate · 12 min read · 2210 words · By Sandip Das · Published
Public postmortem: based on what the organisation published. Sources are listed below.
What this covers
- Background
- What Happened
- Root Cause
- Detection and Response
- Impact
- The Fix
- Lessons for Your Team
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Network availability during outage | Normal operation | 85% of network returning errors |
| Outage duration | N/A | 49 minutes (09:47 to ~10:35 UTC) |
| Time to detection | N/A | ~11 minutes after outage began |
| Time from detection to fix identified | N/A | ~29 minutes |
| Latent bug dwell time in production | N/A | 27 days (May 12 to June 8) |
Key components
- VCL (Varnish Configuration Language) — The domain-specific language used to define routing and caching logic at Fastly's edge; the customer configuration that triggered the bug was written in VCL.
- Fastly Edge POPs — Globally distributed Points of Presence that execute VCL logic and serve cached content; the crash propagated across 85% of these simultaneously.
- Configuration Distribution Pipeline — The system that compiles and pushes customer VCL configurations to all POPs; the absence of staged rollout meant the bug was activated globally and instantly.
- Latent Bug (May 12 software update) — A defect introduced in a Fastly software update 27 days before the outage that lay dormant until a specific valid customer configuration activated it.
- Synthetic / External Monitoring — The mechanism (or lack thereof) for detecting availability failures from outside the system; the 11-minute detection gap suggests room for improvement here.
- Canary Deployment for Config Changes — A staged rollout strategy that would have limited the blast radius of the configuration-triggered bug to a small fraction of traffic before global promotion.
Background
Fastly is a major content delivery network (CDN) and edge cloud provider whose infrastructure sits in the critical path for a significant portion of global web traffic. On June 8, 2021, Fastly's network became the single point of failure for an enormous slice of the internet. Customers affected included the UK government's GOV.UK portal, Reddit, Twitch, GitHub, Stack Overflow, the New York Times, the Guardian, the Financial Times, and Amazon, among hundreds of others.
Fastly's edge network operates by running customer-defined routing and caching logic written in Varnish Configuration Language (VCL) — a domain-specific language derived from Varnish Cache. Customers upload VCL configurations through Fastly's API, and those configurations are compiled and distributed to Fastly's globally distributed Points of Presence (POPs). The system is designed so that customer configuration changes are a routine, self-service operation.
At the scale Fastly operates, a latent defect in the software that processes these configurations — one that can be triggered by a specific but valid customer action — is an existential risk to availability. That is exactly what happened on June 8.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Sources
Topics: caching, cdn, incident-response, postmortem, reliability, varnish