Skip to content
sysgeeks

Design Discord's Message Storage, stage 2 of 9: decide

Choose the store

The requirements ask for growth by adding nodes, survival of node loss, and a team too small for constant manual operations.

System so far· 4 parts
1234CLIENTMembersSERVICEAPI serversSERVICEGatewaySERVICEMessagedata service

Select a component to see what it is responsible for and which state it owns.

  1. 1Members → API servers: Send, load history, jump
  2. 2API servers → Gateway: New message event
  3. 3Gateway → Members: Push to online members
  4. 4API servers → Message data service: Query by channel (hash-routed)
  • Request / response
  • Asynchronous
  • Server push

What you need to know

0 of 2 checks done

  1. A log-structured (LSM) store turns every write into an append: new data goes to memory, then is flushed to immutable sorted files on disk, which are periodically merged (compaction). Writes are cheap; reads may consult several files. See Log-structured storage (LSM trees).

  2. A wide-column store like Cassandra or ScyllaDB has two keys per table:

    • The partition key decides which nodes hold a row, and which rows are stored together.
    • The clustering key orders rows within a partition.

    Capacity grows by adding nodes; each partition is replicated (typically 3 copies) automatically. See Partitioning.

  3. Check

    Why not one document per channel holding an array of its messages?