How a New Telemetry Service Took Down OpenAI's Kubernetes API Servers and DNS in December 2024

A new telemetry deployment in December 2024 overwhelmed OpenAI's Kubernetes API servers and DNS, causing a ~4-hour outage across ChatGPT, the API, and Sora.

Containers & K8s · advanced · 12 min read · 1975 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Incident durationServices healthy~4 hours 11 minutes of degradation across ChatGPT, OpenAI API, and Sora
Services affectedAll services operationalChatGPT, OpenAI API, and Sora all simultaneously degraded or unavailable

Key components

  • Kubernetes API Server — Central control-plane component overloaded by the telemetry agent's watch and list calls, causing cluster-wide scheduling and health-check failures.
  • CoreDNS — Cluster DNS service overwhelmed by the volume of DNS queries generated by the new telemetry DaemonSet, breaking service discovery for all application pods.
  • Telemetry DaemonSet — The new observability workload deployed cluster-wide simultaneously, acting as the thundering-herd source that triggered both API server and DNS overload.
  • NodeLocal DNSCache — A Kubernetes add-on that caches DNS at the node level, which would have absorbed the DNS query spike before it reached CoreDNS — a key missing safeguard.
  • etcd — The backing store for the Kubernetes API server; indirectly stressed by the flood of list and watch requests that the API server forwarded to it.
  • DaemonSet Rolling Update Strategy — The rollout mechanism for the telemetry agent; a staged or canary rollout strategy would have limited the blast radius to a subset of nodes.

Background

On December 11, 2024, OpenAI experienced a significant outage affecting ChatGPT, the OpenAI API, and Sora simultaneously. OpenAI operates one of the highest-traffic AI platforms in the world; at the time, ChatGPT alone was serving hundreds of millions of users. The company runs its inference and serving infrastructure on Kubernetes clusters of substantial scale, where the Kubernetes API server and cluster DNS are shared critical-path dependencies for almost every service operation — pod scheduling, service discovery, health checking, and config reloads all funnel through them.

The incident was publicly documented on the OpenAI status page, which remains the primary public record. The postmortem does not disclose specific cluster sizes, node counts, or internal team names.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: kubernetes, dns, telemetry, incident-response, containers, observability

Related case studies

Browse all case studies