How a Config Push and a Maintenance Tool Severed Google Cloud Networking in June 2019

A misconfigured ACL pushed to backbone routers, combined with a maintenance tool that withdrew routes, dropped Google Cloud network capacity across US regions for up to 4 hours 45 minutes.

Networking & APIs · advanced · 12 min read · 2402 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Incident durationFull network capacityDegraded for approximately 4 hours 45 minutes
Geographic scopeAll US regions healthyus-east1, us-central1, us-west1, us-west2 and some non-US regions affected
Services affectedAll GCP services operationalCompute Engine, Cloud Storage, App Engine, Cloud SQL, BigQuery, GKE, Pub/Sub, Dataflow, Stackdriver, and Google consumer products degraded

Key components

  • Backbone Router ACL Configuration — Access control lists applied to backbone router transit interfaces controlled which traffic was forwarded; a misconfigured ACL reduced effective forwarding capacity.
  • Automated Config Management Pipeline — The system used to generate and push router configurations at scale; it propagated the bad ACL to multiple routers before the impact was detected.
  • Maintenance Drain Tool — Automation used to gracefully withdraw BGP routes from routers before maintenance; it operated concurrently with the bad ACL push, compounding capacity loss.
  • Pre-push Validation Checks — Automated tests that run before a config change is applied to production routers; they failed to catch the ACL error or simulate its interaction with the maintenance tool.
  • BGP Route Withdrawal — The mechanism by which the maintenance tool removed routes from affected routers, reducing the set of viable forwarding paths on an already-impaired network.
  • Canary Deployment Gate (post-incident) — A safeguard added after the incident to push config changes to a small subset of routers first and measure traffic impact before proceeding to the full fleet.

Background

On 2 June 2019, Google Cloud suffered one of its most significant networking incidents on record. The failure affected multiple Google Cloud services — including Compute Engine, Cloud Storage, App Engine, Cloud SQL, BigQuery, Kubernetes Engine, and others — primarily in US regions. Because Google's own consumer products (Search, Gmail, YouTube, Google Drive) share substantial infrastructure with Google Cloud, those services were also degraded during the incident window.

Google's network is one of the largest private WANs on the planet. It carries traffic between data centers, to and from peering points, and across its backbone to serve both internal and external customers. The backbone routers that sit at the heart of this network are managed through automated configuration tooling — a standard practice at hyperscaler scale, where no human can manually manage thousands of devices. This automation is exactly what made the incident both possible and severe.

The public postmortem for Incident 19009 is unusually candid for a vendor disclosure. Google described the root cause, the compounding factors, and the specific safeguard failures that turned a bad config push into a multi-hour, multi-region event.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: networking, google cloud, incident response, bgp, configuration management, postmortem

Related case studies

Browse all case studies