Slack's January 2021 Outage: How AWS Transit Gateway Limits Took Down the First Workday of the Year

On Jan 4 2021, Slack went down for ~4.5 hours as returning workers overwhelmed AWS Transit Gateway connection limits, exposing a capacity blind spot in their network architecture.

Networking & APIs · intermediate · 12 min read · 1908 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Outage durationService available~4.5 hours of degraded or unavailable service
ScopeNormal operationsGlobal impact, most severe for North American users on Jan 4 2021

Key components

  • AWS Transit Gateway — Managed network hub routing traffic between Slack's VPCs; its connection limits were the direct cause of the outage.
  • AWS Service Quotas — Account-level limits on managed service usage that must be proactively monitored and increased before they become ceilings.
  • WebSocket reconnection logic — Client-side persistent connection management; without backoff and jitter, simultaneous reconnects create a thundering herd.
  • CloudWatch quota alarms — The missing safeguard: alerting on quota utilisation before it reaches 100% would have given Slack time to act before the outage.
  • Capacity planning for predictable events — The practice of pre-provisioning resources and requesting quota increases ahead of known demand spikes like the first workday of the year.

Background

Slack is a cloud-based team messaging platform used by millions of organisations worldwide. By early 2021 it was one of the most business-critical communication tools on the internet, with usage patterns tightly coupled to working hours. The platform runs on AWS and uses a multi-VPC architecture stitched together by AWS Transit Gateway — a managed service that acts as a regional network hub, routing traffic between VPCs and on-premises networks without requiring a full mesh of VPC peering connections.

Transit Gateway was a relatively modern addition to Slack's infrastructure, adopted to simplify what had become a complex web of VPC peering relationships as the platform scaled. Like many managed AWS services, it comes with per-account, per-region service quotas that are not always visible in standard monitoring dashboards.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: networking, aws, incident-response, capacity-planning, transit-gateway, postmortem

Related case studies

Browse all case studies