How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB
Discord's Cassandra cluster hit latency and hotspot limits at trillions of messages; a full migration to ScyllaDB cut p99 latency and reduced node count from 177 to 72.
Data & Storage · advanced · 12 min read · 2163 words · By Sandip Das · Published
Public postmortem: based on what the organisation published. Sources are listed below.
What this covers
- Background
- What Happened
- Root Cause
- Detection and Response
- Impact
- The Fix
- Lessons for Your Team
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Cluster node count | 177 Cassandra nodes | 72 ScyllaDB nodes |
| Messages stored | Trillions (growing) | Trillions (same dataset, fully migrated) |
| Data loss during migration | N/A | Zero messages lost (per public account) |
Key components
- Apache Cassandra — Original wide-column message store; 177-node cluster that hit JVM GC and partition hotspot limits at trillion-message scale.
- ScyllaDB — C++ reimplementation of Cassandra with per-core shard architecture and no JVM; replaced Cassandra at 72 nodes for the same workload.
- CQL wire-protocol compatibility — ScyllaDB speaks the same Cassandra binary protocol, allowing Discord's application layer to migrate without code changes.
- Rust-based data migrator — Custom service that performed token-range scans on Cassandra and idempotent upserts into ScyllaDB during the dual-write migration window.
- Time-bucketed partition key — The (channel_id, bucket) composite partition key distributed message data across time windows to limit partition size, but did not fully prevent hotspots on large active channels.
- Dual-write strategy — New writes were sent to both clusters simultaneously during migration to keep ScyllaDB current before reads were shifted over.
Background
Discord stores every message ever sent on its platform — private DMs, server channels, and system events — in a single logical messages table. By 2023, that table held trillions of messages and was growing at hundreds of millions of new records per day. The storage system backing this was Apache Cassandra, a choice Discord made early in its life because Cassandra's wide-column model maps naturally onto the channel-partition / message-clustering-key access pattern that Discord's read and write paths require.
For years the system worked. Discord's engineers tuned Cassandra carefully, ran it at scale, and published their learnings. But as the dataset grew past the trillion-message mark, the cracks became impossible to paper over. The cluster had expanded to 177 nodes spread across multiple rings, and the operational complexity, latency tail behaviour, and a persistent hotspot problem were consuming a disproportionate share of engineering time.
This case study follows Discord's public account of why Cassandra stopped being the right answer, what they replaced it with, and how they moved the data without losing a single message.
---
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Sources
Topics: cassandra, scylladb, database-migration, distributed-systems, data-engineering, performance
Related case studies
- How a B2B SaaS Company Turned a Swampy S3 Data Lake into a Lakehouse with Apache Iceberg
- How a B2B SaaS Team Cut Postgres P99 Latency by 73% with Pooling, Replicas and Partitioning
- How a B2B SaaS Team Migrated 8 TB from Self-Managed MySQL to Aurora with Zero Downtime
- How Figma Scaled Postgres from One Overloaded Database to Vertical Partitioning and Horizontal Sharding
- The 2017 GitLab Database Deletion: When Five Backup Methods All Failed at Once
- The GitHub October 2018 Outage: How 43 Seconds of Network Partition Caused 24 Hours of Data Inconsistency