Skip to content
sysgeeks

Design a Payment System, stage 9 of 13: break it

The commit nobody acted on

The payment state is right. What is missing is everything that was supposed to follow from it. The state change and the follow-up actions live in different systems.

System so far· 5 parts
123456CLIENTBuyer's browserSERVICECheckout APIDATABASEPostgresEXTERNALPayment providerSERVICEWebhook receiver

Select a component to see what it is responsible for and which state it owns.

  1. 1Buyer's browser → Payment provider: Card entry and 3-D Secure in hosted fields
  2. 2Buyer's browser → Checkout API: Pay (Idempotency-Key header); poll status
  3. 3Checkout API → Payment provider: Create payment with the attempt's key
  4. 4Checkout API → Postgres: Claim attempt; conditional transitions
  5. 5Payment provider → Webhook receiver: Signed events: at least once, unordered
  6. 6Webhook receiver → Postgres: Dedupe by event id; forward-only transition
  • Request / response
  • Asynchronous

What you need to know

0 of 2 checks done

  1. After a payment succeeds, other things must follow: grant the course, send a receipt. These live in other systems. If your code commits the payment and then calls them, a crash between the two loses the follow-up, and nothing records that it was owed.

    Logging an error doesn't help: a killed process writes no log line.

  2. The transactional outbox records the obligation inside the same transaction as the state change: "grant course for attempt 77" and "send receipt for attempt 77" as rows in an outbox table. Either the payment and its obligations commit together, or neither does.

    A worker then reads outbox rows, performs each action, and marks it done. If it crashes, the row is still there and is retried. See Transactional outbox.

  3. Check

    The outbox worker grants the course, then crashes before marking the row done. What happens next, and what must the handler do about it?