Skip to content
sysgeeks

Design a Video Processing Pipeline, stage 7 of 14: break it

A worker dies 14 minutes in

Job 812 is marked running with the dead worker as its owner. Nothing will ever touch it again unless the design makes something do so. The requirements say a crashed or redeployed worker must not strand a job.

System so far· 8 parts
123456789CLIENTInstructorbrowserSERVICEVideo APIDATABASEPostgresOBJECT STOREObject storageWORKERTranscodeworkersWORKERReconcilerEDGECDNCLIENTStudent player

Select a component to see what it is responsible for and which state it owns.

  1. 1Instructor browser → Video API: Create upload, report parts, poll status
  2. 2Instructor browser → Object storage: Upload parts via presigned URLs
  3. 3Video API → Object storage: Complete multipart upload, verify object
  4. 4Video API → Postgres: Video row and job row in one transaction
  5. 5Transcode workers → Postgres: Claim lease, heartbeat, fenced completion
  6. 6Transcode workers → Object storage: Read raw upload, write attempt output
  7. 7Reconciler → Postgres: Find abandoned uploads and orphaned output
  8. 8CDN → Object storage: Origin fetch on cache miss
  9. 9Student player → CDN: Manifest and segments
  • Request / response
  • Bulk data

What you need to know

0 of 2 checks done

  1. A process that crashes can't report that it crashed. Out-of-memory kills, hardware failures and hard kills after a shutdown grace period never run cleanup code.

    So recovery from a crashed worker can't depend on the worker doing anything.

  2. A lease is ownership with an expiry: the worker owns the job until leased_until. While it works, it heartbeats, pushing leased_until forward every so often. If it dies, the heartbeats stop, the lease runs out, and the job becomes claimable again.

    Use the database's clock (now()) for expiry, so machines with skewed clocks agree on when a lease ends. See Leases and fencing tokens.

  3. Check

    Jobs take up to 20 minutes. Why use a 2-minute lease with 30-second heartbeats instead of a 25-minute lease?