The AWS us-east-1 outage of October 20, 2025 was a DNS failure
An empty DNS record for DynamoDB's regional endpoint took down much of us-east-1 for about 15 hours. What failed, and what a second DNS provider would and would not have changed.
Published
On October 20, 2025, a large part of AWS’s us-east-1 region stopped working. Slack, Atlassian, Snapchat and more than 70 other services had errors for hours, and some AWS services did not fully recover until the afternoon (Pacific time). The trigger was DNS.
What happened
AWS’s post-event summary describes a latent race condition in the automation that manages the DNS records of DynamoDB’s regional endpoint, dynamodb.us-east-1.amazonaws.com. That automation has a planner that produces DNS plans and an enactor, run redundantly in three Availability Zones, that applies them to Route 53.
Two enactors applied plans at the same time. One of them applied an old plan, and the cleanup of old plans then deleted it, leaving the endpoint’s record empty: no IP addresses at all. The automation could not repair it by itself.
The timeline, in Pacific time:
- 11:48 PM, October 19: DynamoDB API errors start.
- 12:38 AM: the DNS state is identified as the cause.
- 2:25 AM: the DNS information is restored.
- 2:40 AM: customers can resolve the endpoint again.
- The rest of the day: services that depend on DynamoDB (EC2 instance launches, Lambda, load balancers and others) work through their own backlogs. AWS reports EC2 APIs and launches normal at 1:50 PM and the full recovery at 2:20 PM.
What it was not
It was not a failure of Route 53 serving customers’ zones. The record lived in Route 53, and Route 53 served exactly what it was given: an empty answer. The failure was in the system that decides what the record should contain.
That distinction matters for anyone thinking about DNS redundancy.
How a two-provider zone behaves here
A zone served by two providers, which is what RedundantDNS runs, protects you against one kind of failure: a provider that stops answering. If Route 53 had been unreachable, resolvers would have retried the nameservers of the second provider and kept getting answers.
It does not protect you against wrong data. RedundantDNS applies the canonical zone to every provider and then verifies that each one matches it. If the canonical zone says the record is empty, both providers will faithfully serve an empty record, just as Route 53 did. Redundancy copies mistakes as reliably as it copies good records.
What helps against wrong data is different and worth having anyway:
- A change journal. Every change to a RedundantDNS zone is written to the zone’s journal and the audit log (who, what, when, from which client), so you can see the last good version and put it back.
- Probes that look at answers, not just reachability. Probes query every nameserver from each probe region and compare the answers with the zone. They tell you when a provider drifts from the zone. They cannot tell you that the zone itself is wrong; a human or a test has to.
- Fewer robots with write access. An API token is scoped (
zones:read,zones:write) and can carry an IP allowlist, so a script can read everything and still be unable to change records.
The honest summary: October 20 was an argument for treating DNS as critical infrastructure, and for keeping more than one provider in the NS set. It was not an outage that a second provider alone would have prevented.