The 2017 Amazon S3 Outage: One Mistyped Command and the Cost of a Slow Restart

A mistyped parameter in an S3 subsystem maintenance command removed too many servers, triggering a cascade that took down S3 in us-east-1 for ~4 hours and disrupted a large portion of the internet.

Infrastructure · beginner · 12 min read · 2229 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
S3 outage duration in us-east-1Fully operationalApproximately 4 hours 5 minutes of degradation or unavailability
AWS services impactedNormal operationEC2 console, Lambda, Elastic Beanstalk, CloudWatch, and others degraded
AWS Service Health DashboardFully functionalPartially impaired during the incident due to S3 dependency

Key components

  • S3 Index Subsystem — Managed metadata for S3 objects and buckets; its removal caused immediate loss of S3 read and write capability.
  • S3 Placement Subsystem — Managed the physical location of objects across S3 storage nodes; depended on the Index subsystem and failed alongside it.
  • Fleet Management Runbook Tool — The internal operational tool used to remove servers from subsystem pools; lacked a capacity-percentage guard that would have prevented the over-removal.
  • AWS Service Health Dashboard — The customer-facing status page that was itself impaired during the outage because it depended on S3-hosted assets.
  • Subsystem Restart Process — The recovery path for the Index subsystem; had not been exercised in years, causing unexpectedly long state rebuild time and extending the outage.
  • Capacity Removal Rate Limiter (post-fix) — The safeguard added after the incident to prevent any single command from removing more than a defined percentage of a subsystem's capacity.

Background

Amazon S3 is the object storage backbone of a significant fraction of the internet. In February 2017, it had been running in production for over a decade, storing trillions of objects and serving millions of requests per second across AWS regions. The us-east-1 region is the oldest and most heavily used AWS region, and many AWS services — including the EC2 management console, Lambda, Elastic Beanstalk, and CloudWatch — depend on S3 in that region to serve their own assets, configuration files, and deployment packages.

Because of this deep dependency graph, an S3 outage in us-east-1 is not just a storage outage. It is effectively a partial internet outage. Ironically, even the AWS Service Health Dashboard itself relied on S3-hosted assets to render correctly, which meant that during the incident, the dashboard that customers checked for status updates was also degraded.

This case is important not because the failure was exotic — it was not. It is important because it demonstrates how a single operational mistake, in a system with no safeguard against it, can cascade through a massively distributed architecture in minutes.

---

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: aws, s3, outage, infrastructure, incident-response, reliability

Related case studies

Browse all case studies