The Facebook October 2021 Outage: How One BGP Withdrawal Took Down 3.5 Billion Users

A routine backbone maintenance command withdrew Facebook's BGP routes and silenced its DNS servers, making every service unreachable for ~6 hours and locking engineers out of the tools needed to fix it.

Networking & APIs · intermediate · 12 min read · 2364 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Outage durationFull service available~6 hours of complete unavailability for all Facebook family apps
BGP prefixes visible to internetAll Facebook IP prefixes advertised globallyZero Facebook prefixes in global BGP routing tables for ~6 hours
Services affectedFacebook, Instagram, WhatsApp, Messenger, Oculus all operationalAll five services completely unreachable worldwide

Key components

  • BGP (Border Gateway Protocol) — The routing protocol Facebook uses to advertise its IP prefixes to the internet; withdrawal of these announcements made every Facebook IP unreachable.
  • Authoritative DNS servers — Facebook's own DNS infrastructure, hosted inside the data-centre network, which became unreachable when the backbone went down, preventing any client from resolving Facebook hostnames.
  • Backbone network — The private fibre network connecting Facebook's data centres; a maintenance command accidentally took down all backbone links, triggering the cascade.
  • Health-check-driven BGP withdrawal — An automated system that withdraws BGP route advertisements when DNS servers are detected as unreachable — correct in partial failure, catastrophic when all paths fail simultaneously.
  • Out-of-band management access — The physical and logical path engineers use to reach routers when the primary network is down; its absence or slowness extended the outage to six hours.
  • Control plane / data plane separation — The architectural principle that management tooling must not depend on the infrastructure it manages; its violation meant engineers lost their tools at the same moment the network failed.

Background

On 4 October 2021, Facebook, Instagram, WhatsApp, Messenger, and Oculus went completely dark for approximately six hours. At the time, Facebook reported roughly 3.5 billion monthly active users across its family of apps. The outage was not a DDoS attack, not a software bug in the traditional sense, and not a cloud-provider failure. It was a self-inflicted wound caused by a single maintenance command that withdrew every BGP route Facebook had advertised to the internet — and, in doing so, severed the internal network paths that engineers needed to diagnose and fix the problem.

Facebook runs its own global backbone network connecting its data centres, and it advertises its IP prefixes to the internet via BGP (Border Gateway Protocol). Its authoritative DNS servers — the machines that translate facebook.com into IP addresses — sit inside that network. When the backbone became unreachable, so did DNS, and with DNS gone, no client anywhere on the internet could resolve a Facebook hostname. The two Meta Engineering blog posts published on 4 and 5 October 2021 are the primary public record of what happened, and this case study draws exclusively from those sources.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: networking, bgp, dns, incident-response, infrastructure, postmortem

Related case studies

Browse all case studies