The GitHub October 2018 Outage: How 43 Seconds of Network Partition Caused 24 Hours of Data Inconsistency

A 43-second network partition between GitHub's US East Coast data centers caused Orchestrator to promote a new MySQL primary, splitting writes across two nodes and triggering 24 hours of degraded service.

Data & Storage · advanced · 12 min read · 2062 words · By Sandip Das · Published

Public postmortem: based on what the organisation published. Sources are listed below.

What this covers

  1. Background
  2. What Happened
  3. Root Cause
  4. Detection and Response
  5. Impact
  6. The Fix
  7. Lessons for Your Team
  8. Apply It Yourself

Key numbers

MetricBeforeAfter
Incident durationNormal operation24 hours 11 minutes of degraded service
Network partition duration that triggered the incidentN/A43 seconds
Services affectedAll services nominalGitHub.com, Pages, Actions (beta), and API all degraded

Key components

  • Orchestrator — Open-source MySQL topology manager that automatically promoted a replica during the partition, creating the split-brain condition.
  • MySQL primary-replica replication — The data persistence layer whose asynchronous replication allowed two nodes to diverge once both believed they were the writable primary.
  • GTID (Global Transaction Identifiers) — MySQL feature that enabled engineers to precisely identify divergent transactions between the two primaries and guide manual reconciliation.
  • Multi-datacenter network link — The inter-DC connection whose 43-second interruption triggered the automated failover that caused the incident.
  • Raft consensus (post-incident) — The quorum-based decision mechanism added to Orchestrator so that a single observer on one side of a partition can no longer unilaterally trigger a promotion.

Background

GitHub is one of the world's largest software development platforms, hosting tens of millions of repositories and serving millions of developers daily. Its core data layer at the time of this incident relied on MySQL with a primary-replica replication topology spanning two geographically separated data centers on the US East Coast. Orchestrator, an open-source MySQL topology manager originally developed at GitHub, was responsible for automatically detecting primary failures and promoting replicas to maintain availability.

This architecture is common across the industry: a single writable primary, multiple read replicas, and an automated failover tool watching the topology. The October 2018 incident exposed a subtle but catastrophic failure mode in this pattern — one that can affect any team running automated failover without the safeguards to prevent a split-brain scenario.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Sources

Topics: mysql, replication, high-availability, orchestrator, incident-response, data

Related case studies

Browse all case studies