Case Studies

Page 3 of the LearnXops case study library: real-world cloud, DevOps, MLOps and AI engineering postmortems and production scenarios.

  1. How One Platform Team Tamed 30 Product Teams on a Single Kubernetes Cluster

    A B2B SaaS company's platform team built a multi-tenant Kubernetes platform using namespaces, Kyverno, and Crossplane, cutting provisioning time from days to minutes and halving incident rate.

    Containers & K8s · advanced · 12 min read

  2. How Request Coalescing Stopped a Cache Stampede After Redis Failover

    A Redis primary failure triggered a stampede that took the API down for 11 minutes. Request coalescing with a singleflight pattern cut thundering-herd errors to zero.

    Caching · intermediate · 12 min read

  3. Knight Capital 2012: How an Incomplete Deployment and a Reused Flag Lost $460 Million in 45 Minutes

    A stale deployment flag reactivated dead code on one of eight servers, causing Knight Capital's trading system to flood markets with unintended orders and lose $460M in 45 minutes.

    CI/CD & Delivery · beginner · 12 min read

  4. Lift-and-Shift to AWS First, Then Modernise: How a B2B SaaS Company Escaped the Data Centre

    A 120-engineer B2B SaaS company migrated 47 workloads from ageing on-prem hardware to AWS in 14 months, cutting P99 latency by 60% and infrastructure spend by 34% after a deliberate two-phase approach.

    Infrastructure · intermediate · 12 min read

  5. Slack's January 2021 Outage: How AWS Transit Gateway Limits Took Down the First Workday of the Year

    On Jan 4 2021, Slack went down for ~4.5 hours as returning workers overwhelmed AWS Transit Gateway connection limits, exposing a capacity blind spot in their network architecture.

    Networking & APIs · intermediate · 12 min read

  6. The $30,000 NAT Gateway Bill: Finding the Traffic and Fixing It with VPC Endpoints

    A B2B SaaS company discovered half its AWS bill was NAT Gateway data charges from S3 and DynamoDB traffic. VPC endpoints eliminated the cost in two weeks.

    Networking & APIs · beginner · 12 min read

  7. The 2-Million-Message SQS Backlog: Beating Poison Messages with DLQs and Visibility Timeouts

    A B2B SaaS team watched 2 million SQS messages pile up overnight because three malformed events kept recycling. Fixing visibility timeouts and wiring a dead-letter queue cleared the backlog in four hours.

    Messaging & Events · beginner · 12 min read

  8. The 2017 Amazon S3 Outage: One Mistyped Command and the Cost of a Slow Restart

    A mistyped parameter in an S3 subsystem maintenance command removed too many servers, triggering a cascade that took down S3 in us-east-1 for ~4 hours and disrupted a large portion of the internet.

    Infrastructure · beginner · 12 min read

  9. The 2017 GitLab Database Deletion: When Five Backup Methods All Failed at Once

    A tired engineer ran rm -rf on the wrong server, wiping GitLab.com's production PostgreSQL data. Five separate backup systems all failed to produce a usable restore, leaving 6 hours of data permanently lost.

    Data & Storage · intermediate · 12 min read

  10. The 73-Hour Roblox Outage: How Consul Streaming and BoltDB Contention Took Down 50 Million Users

    In October 2021, a Consul misconfiguration triggered BoltDB write contention that cascaded into a 73-hour total outage for Roblox, affecting 50M+ daily users. The lesson: service-mesh health depends on the databases underneath it.

    Distributed Systems · advanced · 12 min read

  11. The Atlassian April 2022 Outage: How a Maintenance Script Deleted 883 Customer Sites

    A misconfigured maintenance script permanently deleted 883 Atlassian cloud sites; restoring them took up to 14 days due to brittle restore tooling and manual processes.

    Infrastructure · intermediate · 12 min read

  12. The Cloudflare July 2019 Outage: How One Regex Took Down a Global CDN

    A single catastrophic backtracking regex in Cloudflare's WAF rules consumed 100% CPU across every PoP worldwide, dropping 82% of HTTP traffic for 27 minutes.

    CI/CD & Delivery · intermediate · 12 min read

  13. The Cloudflare June 2022 Outage: How One BGP Change Took 19 Data Centres Offline

    A network change to consolidate IP prefixes withdrew routes from 19 Cloudflare PoPs, making ~50% of Cloudflare's network unreachable for ~35 minutes. The lesson: always stage network changes and validate reachability before cutting over.

    Networking & APIs · intermediate · 12 min read

  14. The Cloudflare November 2025 Outage: How a Database Permission Change Broke Global Proxy Traffic

    A database ACL change caused a config file to double in size, crashing Cloudflare's proxy on 132 PoPs for 66 minutes. The lesson: schema-level gates matter as much as code review.

    CI/CD & Delivery · advanced · 12 min read

  15. The CrowdStrike July 2024 Outage: How a Single Content Update Crashed 8.5 Million Windows Machines

    A logic error in a CrowdStrike Falcon sensor content update caused 8.5 million Windows hosts to BSOD on boot, exposing the danger of pushing untested kernel-level updates without staged rollouts.

    CI/CD & Delivery · intermediate · 12 min read

  16. The Datadog March 2023 Outage: How an Automatic OS Update Severed Networking Across Tens of Thousands of Nodes

    An automatic Linux security patch changed a kernel networking default, silently breaking connectivity across Datadog's multi-region fleet and taking down their own monitoring platform for hours.

    Infrastructure · intermediate · 12 min read

  17. The December 2021 AWS us-east-1 Outage: How an Automated Scaling Event Congested the Internal Network

    An automated scaling activity in AWS us-east-1 triggered internal network congestion that cascaded into widespread service degradation across dozens of AWS services for roughly 7 hours.

    Networking & APIs · advanced · 12 min read

  18. The Facebook October 2021 Outage: How One BGP Withdrawal Took Down 3.5 Billion Users

    A routine backbone maintenance command withdrew Facebook's BGP routes and silenced its DNS servers, making every service unreachable for ~6 hours and locking engineers out of the tools needed to fix it.

    Networking & APIs · intermediate · 12 min read

  19. The Fastly June 2021 Outage: How One Customer Config Triggered a Global CDN Collapse

    A single customer's valid configuration change activated a latent bug in Fastly's VCL software, taking 85% of the network offline for 49 minutes and knocking out major internet properties worldwide.

    Caching · intermediate · 12 min read

  20. The GitHub October 2018 Outage: How 43 Seconds of Network Partition Caused 24 Hours of Data Inconsistency

    A 43-second network partition between GitHub's US East Coast data centers caused Orchestrator to promote a new MySQL primary, splitting writes across two nodes and triggering 24 hours of degraded service.

    Data & Storage · advanced · 12 min read

  21. The November 2023 Cloudflare Outage: When a Power Failure Took Down the Control Plane

    A data centre power failure in November 2023 cascaded into a multi-hour Cloudflare control-plane and analytics outage, exposing hidden dependencies between the data plane and a single facility.

    Infrastructure · advanced · 12 min read

  22. The October 2025 DynamoDB Outage: How a DNS Automation Race Condition Took Down US-EAST-1

    A race condition in AWS's internal DNS automation triggered a cascade that made DynamoDB unavailable in us-east-1 for several hours, taking down thousands of dependent services with it.

    Distributed Systems · advanced · 12 min read

  23. The Pi Day Outage: How a Renamed Kubernetes Node Label Took Reddit Down for Five Hours

    A Kubernetes version upgrade silently renamed a built-in node label, breaking pod scheduling across Reddit's fleet and causing a five-hour site-wide outage on Pi Day 2023.

    Containers & K8s · intermediate · 12 min read

  24. When Lambda Got Expensive: Moving a High-Throughput API from Lambda to ECS Fargate

    A B2B SaaS company's Lambda-based API hit cold-start and cost ceilings at scale; migrating to ECS Fargate cut monthly compute costs by 61% and p99 latency by 74%.

    Architecture · intermediate · 12 min read