Design a Distributed Cache (Memcache), stage 2 of 9: break it
A stale value that never leaves
Two web servers, A and B, touched the same key around the same time. Find the lines that explain why the cache holds the old value indefinitely.
System so far· 5 parts
Select a component to see what it is responsible for and which state it owns.
- 1Users → Web servers: Page request
- 2Web servers → mcrouter: get / multiget, delete
- 3mcrouter → memcached pool: Keys by consistent hash
- 4Web servers → MySQL: Query on miss; writes
What you need to know
A reader that misses does two things at two different times: it reads the database, then later sets the cache. Anything can happen in between, including a write and its delete.
If the write's delete arrives before the reader's set, the delete has nothing to remove, and the set then stores a value read before the write.
Check
The stale value 'Ann' is now in the cache, with no TTL. How long does it stay?The fix is a token: on a miss, the cache hands the reader a lease token for that key. A delete for the key invalidates outstanding tokens. The reader's set must carry its token, and the cache rejects sets whose token was invalidated.
The idea is the same as a fencing token: a write is accepted only if nothing newer has happened since it was authorised.
Think first
Would a 1-hour TTL on every key fix the stale set?