The Datadog March 2023 Outage: How an Automatic OS Update Severed Networking Across Tens of Thousands of Nodes
An automatic Linux security patch changed a kernel networking default, silently breaking connectivity across Datadog's multi-region fleet and taking down their own monitoring platform for hours.
Infrastructure · intermediate · 12 min read · 1977 words · By Sandip Das · Published
Public postmortem: based on what the organisation published. Sources are listed below.
What this covers
- Background
- What Happened
- Root Cause
- Detection and Response
- Impact
- The Fix
- Lessons for Your Team
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Incident duration | Services healthy | ~12-13 hours of degraded or broken connectivity across multiple regions |
| Regions affected | All regions healthy | Multiple AWS regions impacted simultaneously |
| Services affected | Full platform availability | Metrics ingestion, APM, logs, and dashboards degraded or unavailable for a significant portion of customers |
Key components
- unattended-upgrades — Debian/Ubuntu mechanism that automatically applies security packages; delivered the rp_filter change to the entire fleet without staged rollout.
- net.ipv4.conf.all.rp_filter — Linux kernel Reverse Path Filter parameter; when set to strict mode (1) it silently drops packets on hosts with asymmetric routing.
- Multi-region fleet — Datadog's compute nodes spanning multiple AWS regions; the simultaneous update meant no region was available as a healthy reference point.
- Internal network fabric — The overlay or physical network interconnecting Datadog's service nodes; packet drops here cascaded into service-level failures across intake, metrics, and dashboards.
- Staged rollout / canary process — The absent safeguard: a controlled patch wave that would have exposed the rp_filter regression on a small node subset before fleet-wide impact.
Background
Datadog is a SaaS observability platform used by thousands of engineering teams to monitor infrastructure, applications, and logs. As of early 2023 the platform ran across multiple AWS regions and processed enormous volumes of metrics, traces, and log data continuously. The scale matters here: Datadog is not just a product that suffered an outage — it is the monitoring tool that many of its own customers rely on to detect *their* outages. When Datadog goes down, the blast radius extends well beyond Datadog's own engineering team.
On March 8, 2023, Datadog experienced a multi-region infrastructure connectivity incident that degraded or interrupted service for a significant portion of its customer base. The root cause was not a bad deploy, a misconfigured load balancer, or a cloud provider failure. It was an automatic Linux OS security update that silently changed a kernel networking parameter on a large portion of the fleet.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Sources
Topics: infrastructure, linux, networking, incident-response, patch-management, postmortem
Related case studies
- How 37signals Saved $10M by Leaving AWS for Owned Hardware
- How a 120-Engineer SaaS Company Migrated from One AWS Account to a Multi-Account Landing Zone
- How a B2B SaaS Team Cut CI and Batch Costs by 70% Using Spot Instances Without Losing Jobs
- How a B2B SaaS Team Cut Their AWS Bill by 40% with Rightsizing, Graviton, Savings Plans and S3 Lifecycle Policies
- How a B2B SaaS Team Split a 4,000-Resource Terraform Monolith Into Modules Without Downtime
- Lift-and-Shift to AWS First, Then Modernise: How a B2B SaaS Company Escaped the Data Centre