The November 2023 Cloudflare Outage: When a Power Failure Took Down the Control Plane

A data centre power failure in November 2023 cascaded into a multi-hour Cloudflare control-plane and analytics outage, exposing hidden dependencies between the data plane and a single facility.

Infrastructure · advanced · 12 min read · 2326 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Control plane availabilityFully operationalSeverely degraded for several hours on 2 November 2023
Data plane trafficNormalContinued to serve traffic for most customers throughout the outage
Analytics dataReal-timeGap in analytics data for the outage window; pipelines required backlog replay after recovery

Key components

  • Control Plane — Hosts the customer API, dashboard, configuration stores, and orchestration systems; was the primary affected layer.
  • Data Plane — Handles actual customer traffic at edge PoPs; survived the outage by operating on locally cached configuration.
  • UPS and Facility Power — Uninterruptible power supplies engaged as designed but could not bridge the full duration of the power failure.
  • Analytics Pipelines — Ingest and process telemetry from edge nodes; went offline with the control plane and required backlog replay on recovery.
  • Configuration Store — Stateful backend holding zone and rule configuration; its unavailability blocked all customer configuration changes.
  • Readiness and Liveness Probes — Health-check mechanisms that determine whether a service should receive traffic; must probe dependencies to be meaningful.

Background

Cloudflare operates one of the largest edge networks in the world, with over 300 points of presence and a stated mission of making the internet faster and more secure. The company's architecture divides into two broad planes: the data plane, which handles actual customer traffic at the edge, and the control plane, which manages configuration, orchestration, analytics, and the customer-facing dashboard and API.

For most customers, the data plane is what matters second-to-second — it proxies HTTP traffic, applies WAF rules, and terminates TLS. The control plane is what matters when you need to *change* something: update a DNS record, deploy a new firewall rule, read your traffic analytics, or provision a new zone. On 2 November 2023, a power event at a single data centre facility took the control plane offline and demonstrated just how many things depend on it.

The public postmortem is candid about the scope: the data plane continued to serve traffic for most customers throughout the incident, but the control plane — the API, the dashboard, and analytics pipelines — was severely degraded or unavailable for several hours.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: cloudflare, infrastructure, control-plane, postmortem, resilience, power-failure

Related case studies

Browse all case studies