Design a Notification System, stage 6 of 11: break it
Why users got two emails
The notification row is unique. The duplicate happened later, in sending. Find the design decisions that produced it.
System so far· 10 parts
Select a component to see what it is responsible for and which state it owns.
- 1Product services → Event stream: Domain events via outbox
- 2Event stream → Notification planner: Events, at least once
- 3Notification planner → Notifications DB: Insert notifications under dedupe keys
- 4Notification planner → Delivery queues: Deliveries by priority class
- 5Channel senders → Delivery queues: Claim deliveries
- 6Channel senders → Email provider: Send within quota
- 7Channel senders → Push services: Send to each device
- 8Web and mobile apps → Inbox API: Inbox and read state
- 9Inbox API → Notifications DB: Read notifications
- Asynchronous
- Request / response
What you need to know
A queue's visibility timeout is a lease: once a sender receives a message, the queue hides it for that long. If the sender doesn't acknowledge in time, the message reappears and another sender can receive it.
So the visibility timeout has to be longer than the slowest the work can take, or the sender has to extend it while it works.
Check
The visibility timeout is 5 s. The provider call's own timeout is 30 s. What can happen?A second safeguard: record state before the side effect. Before calling the provider, the sender changes the delivery from
queuedtosendingwith a fresh attempt token, in one conditional update:UPDATE deliveries SET status = 'sending', attempt_token = $2 WHERE id = $1 AND status = 'queued'If another sender already claimed it, zero rows change and this one skips the delivery.
Think first
With a long enough lease and the claim, can a duplicate email still happen?