The 2017 GitLab Database Deletion: When Five Backup Methods All Failed at Once

A tired engineer ran rm -rf on the wrong server, wiping GitLab.com's production PostgreSQL data. Five separate backup systems all failed to produce a usable restore, leaving 6 hours of data permanently lost.

Data & Storage · intermediate · 12 min read · 2027 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Downtime durationSite online~18 hours offline
Data permanently lost0 hours~6 hours of production writes
Backup methods that produced a usable restore5 methods configured0 of 5 worked; restore came from an ad-hoc manual dump

Key components

  • PostgreSQL WAL-E archiving — Continuous WAL shipping to object storage — configured but never restore-tested, so it failed when needed.
  • pg_dump scheduled backup — Periodic logical backup — was not running on the production primary at the time of the incident.
  • LVM snapshots — Block-level point-in-time snapshots of the database volume — had been disabled.
  • Streaming replication — db2 replica of db1 — replicated the deletion rather than protecting against it, illustrating that replication is not a backup.
  • Automated restore testing — The safeguard that was entirely absent — no backup method had ever been validated with an end-to-end restore test.
  • Break-glass access controls — Mechanism to require additional confirmation or logging for destructive operations on production database hosts — did not exist.

Background

GitLab.com is the SaaS offering of GitLab, a self-hosted and cloud-hosted Git repository platform. In early 2017 it was already a major platform for open-source and commercial software development, hosting millions of repositories. The database backend was PostgreSQL, running in a primary-replica configuration. GitLab operates with a high degree of public transparency — they publish postmortems, keep public issue trackers, and even streamed parts of their incident response live on YouTube. That transparency is why this incident is one of the most thoroughly documented production disasters in public DevOps history.

The event happened on the night of January 31 / morning of February 1, 2017 UTC. The official postmortem and the live incident notes together give an unusually complete picture of what went wrong.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: postgresql, backup, incident-response, data-loss, postmortem, database

Related case studies

Browse all case studies