The Pi Day Outage: How a Renamed Kubernetes Node Label Took Reddit Down for Five Hours

A Kubernetes version upgrade silently renamed a built-in node label, breaking pod scheduling across Reddit's fleet and causing a five-hour site-wide outage on Pi Day 2023.

Containers & K8s · intermediate · 12 min read · 1852 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Outage durationSite available~5 hours of degraded/unavailable service
Pods affectedScheduled and runningStuck in Pending state across the fleet
Root cause identification timeN/AMultiple hours into the incident before cause was isolated

Key components

  • beta.kubernetes.io/* node labels — Deprecated Kubernetes node labels that were silently removed in the upgraded version, breaking nodeAffinity rules fleet-wide.
  • nodeAffinity requiredDuringSchedulingIgnoredDuringExecution — The scheduling constraint that caused pods to enter Pending state when their label selectors could no longer match any node.
  • kube-scheduler — The Kubernetes component that evaluated nodeAffinity rules and was unable to place pods once the referenced labels disappeared from nodes.
  • Kubernetes version upgrade — The planned maintenance event that triggered the label removal and exposed the stale manifest references.
  • Deployment manifests — The fleet of YAML manifests containing stale beta.kubernetes.io/* label references that were never migrated to stable equivalents.
  • Pluto / manifest scanning — Post-incident tooling recommendation for detecting deprecated API versions and label references before an upgrade is executed.

Background

Reddit is one of the largest social platforms on the internet, serving hundreds of millions of users and handling enormous traffic spikes around cultural moments. Its backend runs on Kubernetes, with workloads spread across a substantial cluster fleet managed by an internal platform engineering team. On March 14, 2023 — Pi Day — Reddit's engineering team executed a planned Kubernetes version upgrade. What followed was a five-hour outage that took the site largely offline and became one of the most instructive public Kubernetes postmortems of the year.

The official postmortem was published by Reddit Engineering on their own subreddit, r/RedditEng. It is unusually candid and detailed, and everything factual in this case study is drawn from that document.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: kubernetes, containers, incident-response, node-affinity, postmortem, platform-engineering

Related case studies

Browse all case studies