Skip to content
sysgeeks

Design a Notification System, stage 3 of 11: decide

One notification per event, per channel

The event stream redelivers events after consumer restarts. Two planner instances can process the same event concurrently. Each must produce the same notifications, once.

System so far· 3 parts
12SERVICEProduct servicesLOG / STREAMEvent streamWORKERNotificationplanner

Select a component to see what it is responsible for and which state it owns.

  1. 1Product services → Event stream: Domain events via outbox
  2. 2Event stream → Notification planner: Events, at least once

What you need to know

0 of 2 checks done

  1. Event streams deliver at least once. A consumer that processes an event and crashes before recording its progress will see that event again after restarting. Two consumer instances can also briefly process the same event during a rebalance.

    So the planner must produce the same result however many times it sees an event.

  2. The most robust way is to give each notification an identity derived from its cause: (recipient, event id, channel). Store it under a unique constraint and insert with "ignore on conflict":

    INSERT INTO notifications (dedupe_key, user_id, …)
VALUES ('u88:evt991:email', 88, …)
ON CONFLICT (dedupe_key) DO NOTHING;

    The first insert creates the row. Every replay hits the constraint and does nothing.

  3. Check

    Instead of a unique constraint, the planner first checks a Redis set of processed event ids, then inserts. Two planners get the same event at once. What can happen?