Discord: storing chat history forever
Chat history that is kept forever and read at random, and the move to another database while it was running.
The idea
Chat history looks like a log, but the reads are anything but sequential: people open old channels, jump to search results and scroll back years. Discord's two posts, six years apart, follow one data model through that.
The first explains the model: partition messages by channel plus a time bucket, so no partition grows without bound, and sort them by a time-ordered ID so the latest page is cheap. It also covers what that storage engine makes expensive: a delete leaves a tombstone that later reads must step over, and an edit racing a delete can leave a broken row.
The second covers what growth did to the same design: hot partitions when everyone reads one huge channel at once, fixed by routing every request for a channel through one service instance that merges identical reads. Then the whole history moved to a different database without downtime.
Read the originals
Written by the engineers who built it.
- How Discord Stores Trillions of Messages
Bo Ingram · Post, Mar 2023
The same data model six years on: hot partitions, a service layer that merges identical reads, and a migration of the full history to a new database.
- How Discord Stores Billions of Messages
Stanislav Vishnevskiy · Post, Jan 2017
Choosing a database and a partition key for chat history, and the surprises that followed: tombstones, and an edit racing a delete.
Practise it
Make the decisions yourself, then compare.