The Cloudflare June 2022 Outage: How One BGP Change Took 19 Data Centres Offline

A network change to consolidate IP prefixes withdrew routes from 19 Cloudflare PoPs, making ~50% of Cloudflare's network unreachable for ~35 minutes. The lesson: always stage network changes and validate reachability before cutting over.

Networking & APIs · intermediate · 12 min read · 1898 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
PoPs taken offlineAll 19 PoPs reachable19 PoPs unreachable for ~35 minutes
Network capacity affected100% of network serving~50% of global network degraded
Time to full recoveryN/A~35 minutes from first impact

Key components

  • BGP Anycast Routing — Cloudflare announces the same IP prefixes from hundreds of PoPs; BGP steers users to the nearest healthy site, making prefix withdrawal an instant outage trigger.
  • IP Prefix Specificity (Longest Prefix Match) — BGP prefers more-specific routes; withdrawing a /24 before the /21 replacement is propagated leaves the PoP with no valid route.
  • Network Change Automation — The tooling used to apply the prefix consolidation lacked a mandatory reachability-verification gate between the announce and withdraw steps.
  • Staged Rollout Policy — The absence of a canary-style staged rollout meant the change hit all 19 PoPs simultaneously, maximising blast radius.
  • External Reachability Monitoring — RIPE RIS, BGPmon, or looking glasses at transit peers provide the external vantage point needed to detect that a PoP is unreachable from the internet.
  • Automatic Rollback on Traffic Drop — A post-change traffic-level gate that triggers automatic revert is the safety net that prevents a fast-moving BGP incident from lasting more than seconds.

Background

Cloudflare operates one of the largest anycast networks in the world, spanning hundreds of points of presence (PoPs) across more than 270 cities. As of mid-2022, it handled roughly 10% of all HTTP requests on the internet, proxied DNS for millions of domains via its 1.1.1.1 resolver, and protected customers with its WAF, DDoS mitigation, and Workers serverless platform. The network's reliability is foundational: when Cloudflare's PoPs go unreachable, the customer's origin server is often exposed or entirely unreachable to end users.

The network relies heavily on BGP (Border Gateway Protocol) to advertise IP prefixes to upstream transit providers and peers. Cloudflare uses anycast routing, meaning the same IP address is announced from many PoPs simultaneously; traffic is naturally steered to the nearest healthy PoP by BGP's shortest-path logic. This architecture is both a superpower and a fragility: a misconfigured route advertisement can silently remove a PoP from the internet without touching a single server inside it.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: networking, bgp, incident-response, routing, postmortem, infrastructure

Related case studies

Browse all case studies