How a B2B SaaS Team Built Multi-Region DR on AWS with a Tested 15-Minute RTO
A 120-engineer SaaS company replaced a single-region architecture with active-passive multi-region DR on AWS, cutting RTO from 4+ hours to under 15 minutes through automation, chaos testing, and disciplined runbook engineering.
Distributed Systems · advanced · 12 min read · 2511 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Recovery Time Objective (full regional failure) | 4–6 hours (untested estimate) | 11–13 minutes (3 drills average) |
| Failover automation coverage | 0% — fully manual runbook | 100% automated via Step Functions and SSM |
| Aurora replication lag (steady state) | N/A — single region | Under 1 second (Aurora Global DB) |
| Time to detect regional failure | Manual — no alerting | 30–60 seconds via CloudWatch composite alarm |
| Monthly DR infrastructure cost | $0 — no DR region | $14,200/month warm standby |
Key components
- Aurora Global Database — Provides sub-second cross-region replication of the PostgreSQL data store with a single API call to promote the secondary to a standalone writer during failover.
- AWS Step Functions — Orchestrates the multi-step failover sequence — Aurora promotion, SSM parameter update, Route 53 cutover — with built-in retry, error handling, and execution history.
- Route 53 Failover Routing with Health Checks — Detects primary region failure within 30–60 seconds and shifts DNS to the DR ALB, with TTL tuned to 60 seconds to minimise client-side caching delay.
- SSM Automation Document — Codifies the recovery runbook as a versioned, auditable document that can be executed by Step Functions or manually by an on-call engineer.
- S3 Cross-Region Replication with RTC — Replicates document assets to the DR region with a 15-minute SLA, ensuring object availability after a failover without manual intervention.
- Monthly DR Drill Cadence — Scheduled chaos tests that simulate full regional failure, measure actual RTO, and surface configuration drift before a real incident does.
The Situation
This is a composite scenario assembled from patterns observed across multiple production engineering teams; all figures are illustrative and chosen for internal consistency, not to represent any single organisation.
The organisation is a 120-engineer B2B SaaS company selling workflow-automation software to mid-market financial services firms. Their platform processes roughly 4 million API calls per day and holds sensitive customer data subject to SOC 2 Type II and, for several enterprise accounts, ISO 27001 requirements. The engineering team is split into eight product squads and a 12-person platform engineering group responsible for infrastructure, reliability, and security. The stack is entirely AWS: ECS Fargate for the API tier, Aurora PostgreSQL for the primary data store, ElastiCache Redis for session and rate-limit caching, and S3 for document storage. Everything is deployed via Terraform and released through a GitHub Actions pipeline. At the time the project started, all production workloads lived exclusively in us-east-1.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: disaster-recovery, aws, distributed-systems, rto, infrastructure-as-code, resilience