The Cloudflare November 2025 Outage: How a Database Permission Change Broke Global Proxy Traffic

A database ACL change caused a config file to double in size, crashing Cloudflare's proxy on 132 PoPs for 66 minutes. The lesson: schema-level gates matter as much as code review.

CI/CD & Delivery · advanced · 12 min read · 2026 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
PoPs affected132 PoPs serving traffic normally132 PoPs with proxy process failures for ~66 minutes
Incident durationNormal operation~66 minutes of customer-facing impact
Config file sizeNormal sizeApproximately 2x normal size due to duplicate rows

Key components

  • Config Generation Pipeline — Queries the configuration database and renders nginx-compatible config files for distribution to edge PoPs.
  • Database ACL / Permission Layer — Controls which views and tables the config-generation service account can read; a change here silently altered query results.
  • Config Validator — Checked syntax of generated configs but did not check for semantic duplication or size anomalies, allowing the bad artifact to proceed.
  • PoP Distributor — Pushed the validated (but broken) config to all 132 affected Points of Presence simultaneously.
  • Proxy Process (nginx-compatible) — Refused to load the doubled config due to duplicate server block directives, causing traffic failures.
  • Self-healing Config Pipeline — Automatically regenerated correct configs once the source database was fixed, limiting MTTR to 66 minutes.

Background

Cloudflare operates one of the largest content delivery and reverse-proxy networks on the planet, with hundreds of Points of Presence (PoPs) handling traffic for millions of websites. The system at the centre of this incident is the pipeline that generates, validates, and distributes nginx-style proxy configuration files to every edge PoP. These config files encode routing rules, TLS settings, cache directives, and customer-specific behaviour. Every time a customer changes a DNS record, a firewall rule, or a page rule, the pipeline regenerates the relevant config and pushes it to the appropriate PoPs. Because this pipeline runs continuously and touches the data plane directly, a bug in it does not stay contained — it fans out to every server that receives the bad artifact.

At the scale Cloudflare operates, this pipeline is both a competitive advantage and a critical single point of failure. The November 18, 2025 postmortem is unusually candid about what went wrong and why the safeguards that existed were not enough.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: cloudflare, ci-cd, database, configuration-management, postmortem, reliability

Related case studies

Browse all case studies