Reconciliation
Periodically comparing your records with an authoritative source and repairing differences, the backstop for every message that was lost.
Reliability
Learn it
Your system's view of the world is assembled from messages: API responses, webhooks, queue events. Some get lost, some are duplicated past their dedupe windows, some are misapplied by bugs. Without a way to compare against the truth, those errors are permanent and invisible.
A reconciler periodically asks: does my record match the source of truth?
Two common forms:
- Targeted sweeps: find records stuck in non-terminal states too long (
processingfor over 10 minutes), look each up at the source, and apply the result through the same conditional transitions as every other path, so the reconciler and a late webhook can't conflict. - Full comparisons: compare everything in a period against an authoritative report (a settlement file, a bank statement), in both directions.
- Targeted sweeps: find records stuck in non-terminal states too long (
Check
Why must the reconciler use the same transitions as webhooks and API responses?Rules that keep reconcilers honest:
- Reconcile toward the source of truth for each fact. For money, that's the provider, not your database.
- Some differences can't be decided automatically: a record the provider hasn't heard of might be a request still in flight. Wait out that window, and send what remains to a human.
- Alert on the output. A rising count of corrections means something upstream is broken.
Quick reference
The same ideas, condensed for revision.
How it goes wrong
- Reconciler fights the main path
- It writes state unconditionally and overwrites a newer transition made by a webhook.
- Premature conclusions
- It marks an in-flight request failed because the provider has not recorded it yet.
- Nobody reads the output
- Discrepancies are logged and ignored, so the bug causing them persists.
Instead, consider
- Trust the event stream
- Delivery is transactional end to end, within one system, and loss is impossible by construction.
- Manual review
- Volume is tiny and discrepancies are rare enough for a person to check.
In practice
- Scheduled sweep of stale states
- WHERE status = 'processing' AND updated_at < now() - interval '10 minutes'.
- Settlement file comparison
- Daily batch job diffing provider reports against your ledger.
- Anti-entropy between replicas
- The same idea in distributed databases: compare and repair (Merkle trees).
It assumes
- An authoritative source exists and can be queried by your identifiers.
- The reconciler applies changes through the same idempotent, conditional paths as normal processing.
Explain it in your own words
Where you practise it
Related concepts
- Webhooks
HTTP callbacks from another system: delivered at least once, possibly out of order, possibly never. Handle them as hints, not truth.
- Timeouts and unknown outcomes
A timeout bounds how long you wait. It tells you nothing about what happened, so the operation's outcome becomes unknown.
- State machines for business state
Modelling an entity's lifecycle as explicit states and allowed transitions, enforced with conditional updates so concurrent or stale actors cannot corrupt it.
- Append-only logs
Recording changes as an ordered, immutable sequence of facts, from which current state, history and replicas can be derived.
- Online data migrations
Moving live data to a new schema or store without downtime: write to both, backfill the past, verify, switch reads, then switch writes, with a way back at every step.