The Cloudflare July 2019 Outage: How One Regex Took Down a Global CDN

A single catastrophic backtracking regex in Cloudflare's WAF rules consumed 100% CPU across every PoP worldwide, dropping 82% of HTTP traffic for 27 minutes.

CI/CD & Delivery · intermediate · 12 min read · 2157 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
HTTP traffic droppedNormal global traffic82% of traffic dropped at peak
Duration of significant impactNormal operation~27 minutes (13:42–14:09 UTC)
Geographic scopeNormal operationAll Cloudflare PoPs worldwide simultaneously

Key components

  • PCRE Regex Engine — The backtracking regex engine used by the WAF whose worst-case exponential evaluation time caused 100% CPU saturation globally.
  • WAF Rule CI/CD Pipeline — The automated pipeline that deployed WAF rule updates globally without a performance or execution-time gate.
  • Global WAF Kill-Switch — A pre-built operational lever that disabled the entire WAF across all PoPs, enabling traffic recovery in ~2 minutes after activation.
  • Anycast PoP Network — Cloudflare's globally distributed points of presence that all received the bad rule simultaneously, giving the incident a 100% blast radius.
  • Internal Tooling on Cloudflare Network — Cloudflare's own incident-response tools were routed through the affected network, slowing the response when they became unreachable.

Background

Cloudflare operates one of the largest content delivery and security networks on the internet. As of mid-2019, the company was processing roughly 10% of all internet HTTP requests, serving millions of websites and APIs through a globally distributed anycast network spanning hundreds of points of presence (PoPs). One of its core products is a Web Application Firewall (WAF) that inspects every HTTP request passing through those PoPs and applies a set of managed rules to block malicious traffic — SQL injection, XSS, and similar attack patterns.

Because Cloudflare's network is a shared infrastructure, a defect in any globally deployed component does not affect one customer or one region: it affects everyone, everywhere, simultaneously. That systemic coupling is what made the events of July 2, 2019 so severe.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: waf, regex, incident-response, ci-cd, performance, cloudflare

Related case studies

Browse all case studies