The December 2021 AWS us-east-1 Outage: How an Automated Scaling Event Congested the Internal Network

An automated scaling activity in AWS us-east-1 triggered internal network congestion that cascaded into widespread service degradation across dozens of AWS services for roughly 7 hours.

Networking & APIs · advanced · 12 min read · 2255 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Incident duration (onset to broad recovery)Normal operationsApproximately 7 hours of degraded availability
AWS services impacted in us-east-1All services operationalEC2, Lambda, ECS, EKS, Kinesis, CloudWatch, EventBridge, ELB, and others degraded
Service Health Dashboard lagExpected to reflect real-time statusDashboard lagged significantly behind actual customer impact, acknowledged by AWS

Key components

  • AWS Auto Scaling Service — Initiated the automated scaling activity that triggered the connection surge on the internal network.
  • Internal Network Fabric (us-east-1) — The shared network infrastructure whose connection-tracking capacity was overwhelmed by the scaling event, causing broad congestion.
  • Amazon Kinesis Data Streams — A foundational service impaired by the congestion; its degradation had second-order effects on Lambda async invocations and CloudWatch metric delivery.
  • Amazon CloudWatch — The primary observability service for AWS customers, which was itself degraded during the incident, hampering detection and response.
  • AWS Service Health Dashboard — The public-facing status page that lagged behind actual impact, prompting AWS to commit to communication improvements.
  • Connection storm / thundering herd pattern — The failure mode where simultaneous connection establishment by many new nodes overwhelms stateful network devices, the core mechanism of this incident.

Background

On December 7, 2021, a significant portion of AWS infrastructure in the us-east-1 region experienced degraded performance and availability across a wide range of services. The incident affected not only direct AWS customers running workloads in the region but also a large number of downstream consumer applications and SaaS products that depend on AWS services — including Amazon's own internal services such as the Alexa voice platform, Ring doorbells, and Amazon.com itself.

us-east-1 (Northern Virginia) is AWS's oldest and most heavily utilised region, and it hosts a disproportionate share of globally critical infrastructure. Many AWS control-plane services — the APIs and orchestration layers that manage resources across all regions — are centralised there. This makes us-east-1 failures uniquely broad in their blast radius: a problem in us-east-1 can degrade the ability of customers to manage resources even in other regions.

The public postmortem for this event was published by AWS at aws.amazon.com/message/12721/.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: aws, networking, incident-response, auto-scaling, postmortem, cloud-reliability

Related case studies

Browse all case studies