The October 2025 DynamoDB Outage: How a DNS Automation Race Condition Took Down US-EAST-1
A race condition in AWS's internal DNS automation triggered a cascade that made DynamoDB unavailable in us-east-1 for several hours, taking down thousands of dependent services with it.
Distributed Systems · advanced · 12 min read · 2305 words · By Sandip Das · Published
Public postmortem: based on what the organisation published. Sources are listed below.
What this covers
- Background
- What Happened
- Root Cause
- Detection and Response
- Impact
- The Fix
- Lessons for Your Team
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| DynamoDB availability in us-east-1 | Normal operation | Degraded for approximately 3-4 hours per public postmortem |
| Downstream AWS services affected | Normal operation | Multiple services dependent on DynamoDB also degraded per public postmortem |
Key components
- DNS Automation System — Internal AWS automation that manages DNS records for service endpoints; the race condition in this system was the root cause of the outage.
- Distributed Locking — The mechanism that should have serialized concurrent DNS writes; its failure to correctly prevent simultaneous execution allowed conflicting updates.
- Internal DNS Resolution — The layer that translates service names to network targets; when records were corrupted, DynamoDB clients could not reach valid endpoints.
- DynamoDB Request Router — The component that directs client requests to the correct storage nodes; it became unreachable when DNS records pointed to incorrect targets.
- DNS TTL and Cache Propagation — The time-to-live on DNS records determined how long incorrect records persisted in caches after the fix was applied, extending the recovery window.
- Synthetic DNS Monitoring — A safeguard that was either absent or insufficient; direct monitoring of DNS resolution health would have surfaced the root cause faster than observing DynamoDB error rates.
Background
Amazon DynamoDB is a fully managed, serverless key-value and document database that underpins a significant fraction of AWS-hosted workloads. In us-east-1 — AWS's oldest, largest, and most densely populated region — DynamoDB serves not just external customer applications but also internal AWS services. When DynamoDB becomes unhealthy in us-east-1, the blast radius extends far beyond the customers who directly call its API: services that use DynamoDB for their own control planes, session stores, or coordination layers fail too, creating a cascade that is disproportionate to the initial trigger.
In October 2025, that cascade happened. The root cause was not a capacity event, a hardware failure, or a software bug in DynamoDB itself. It was a race condition in an internal DNS automation system — the kind of subtle concurrency defect that is easy to miss in testing but catastrophic at the scale of a hyperscaler's internal network. The AWS public postmortem provides the authoritative account of what happened.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Sources
Topics: dynamodb, dns, distributed-systems, aws, incident-response, race-condition