Networking & APIs Case Studies

Real-world Networking & APIs case studies: what happened in production, the architecture behind it, the numbers and the lessons to take to your own team.

Browse all case studies

  1. Chasing 5-Second DNS Timeouts in Kubernetes: ndots, Conntrack Races, and NodeLocal DNSCache

    Intermittent 5-second API latency spikes traced to Kubernetes DNS conntrack race conditions and ndots misconfiguration, resolved with NodeLocal DNSCache and search-domain tuning.

    Networking & APIs · advanced · 12 min read

  2. How a Config Push and a Maintenance Tool Severed Google Cloud Networking in June 2019

    A misconfigured ACL pushed to backbone routers, combined with a maintenance tool that withdrew routes, dropped Google Cloud network capacity across US regions for up to 4 hours 45 minutes.

    Networking & APIs · advanced · 12 min read

  3. Slack's January 2021 Outage: How AWS Transit Gateway Limits Took Down the First Workday of the Year

    On Jan 4 2021, Slack went down for ~4.5 hours as returning workers overwhelmed AWS Transit Gateway connection limits, exposing a capacity blind spot in their network architecture.

    Networking & APIs · intermediate · 12 min read

  4. The $30,000 NAT Gateway Bill: Finding the Traffic and Fixing It with VPC Endpoints

    A B2B SaaS company discovered half its AWS bill was NAT Gateway data charges from S3 and DynamoDB traffic. VPC endpoints eliminated the cost in two weeks.

    Networking & APIs · beginner · 12 min read

  5. The Cloudflare June 2022 Outage: How One BGP Change Took 19 Data Centres Offline

    A network change to consolidate IP prefixes withdrew routes from 19 Cloudflare PoPs, making ~50% of Cloudflare's network unreachable for ~35 minutes. The lesson: always stage network changes and validate reachability before cutting over.

    Networking & APIs · intermediate · 12 min read

  6. The December 2021 AWS us-east-1 Outage: How an Automated Scaling Event Congested the Internal Network

    An automated scaling activity in AWS us-east-1 triggered internal network congestion that cascaded into widespread service degradation across dozens of AWS services for roughly 7 hours.

    Networking & APIs · advanced · 12 min read

  7. The Facebook October 2021 Outage: How One BGP Withdrawal Took Down 3.5 Billion Users

    A routine backbone maintenance command withdrew Facebook's BGP routes and silenced its DNS servers, making every service unreachable for ~6 hours and locking engineers out of the tools needed to fix it.

    Networking & APIs · intermediate · 12 min read