Skip to content
sysgeeks

Design a Video Processing Pipeline, stage 8 of 14: break it

Two workers, one job

Leases recover crashed workers. But this worker did not crash. It only lost contact with the database for a while. Read the timeline and select the lines where the design (not the network) is at fault.

System so far· 8 parts
123456789CLIENTInstructorbrowserSERVICEVideo APIDATABASEPostgresOBJECT STOREObject storageWORKERTranscodeworkersWORKERReconcilerEDGECDNCLIENTStudent player

Select a component to see what it is responsible for and which state it owns.

  1. 1Instructor browser → Video API: Create upload, report parts, poll status
  2. 2Instructor browser → Object storage: Upload parts via presigned URLs
  3. 3Video API → Object storage: Complete multipart upload, verify object
  4. 4Video API → Postgres: Video row and job row in one transaction
  5. 5Transcode workers → Postgres: Claim lease, heartbeat, fenced completion
  6. 6Transcode workers → Object storage: Read raw upload, write attempt output
  7. 7Reconciler → Postgres: Find abandoned uploads and orphaned output
  8. 8CDN → Object storage: Origin fetch on cache miss
  9. 9Student player → CDN: Manifest and segments
  • Request / response
  • Bulk data

What you need to know

0 of 1 checks done

  1. A lease can expire while its holder is still running. The holder may not know: it may be paused, or unable to reach the database. You can't prevent this, because from the outside a paused process looks exactly like a dead one.

    What you can do is make the old holder's writes harmless.

  2. Freeze worker A for longer than its lease, then switch on fencing. Each new lease comes with a higher token number, and storage remembers the highest token it has accepted.

    A lease that runs out while its holder is asleep. The lease lasts 10 s and A renews it every 3 s, but A freezes at 2 s (a GC pause, a VM migration, a slow disk). Change how long A is frozen, and whether storage checks fencing tokens.
    12 s
    Worker AWorker BStorage
    1. 0 sWorker ATakes the lease with fencing token 33
    2. 2 sWorker AStalls (GC pause) for 12 s
    3. 10 sLockA's lease expires
    4. 10 sWorker BTakes the lease with fencing token 34
    5. 11 sWorker BWrites the job's result
    6. 11 sStorageAccepts B's write (token 34)
    7. 14 sWorker AResumes, still believes it holds the lease, and writes
    8. 14 sStorageAccepts A's write and overwrites B's result

    Two workers acted as the owner. A's lease expired during the pause, B took over, and storage accepted A's late write anyway. The job's result is now whatever A wrote last.

  3. Check

    With fencing on, worker A resumes and writes with token 33 after worker B has written with token 34. What happens to A's write?