Skip to content
sysgeeks

Design a Video Processing Pipeline, stage 5 of 14: model

Trace the happy path end to end

You have made the structural decisions. Before you start breaking things, trace one video all the way through. Getting the order right matters: several of the system's guarantees depend on which step happens before which.

System so far· 6 parts
1234567CLIENTInstructorbrowserSERVICEVideo APIDATABASEPostgresOBJECT STOREObject storageWORKERTranscodeworkersWORKERReconciler

Select a component to see what it is responsible for and which state it owns.

  1. 1Instructor browser → Video API: Create upload, report parts, poll status
  2. 2Instructor browser → Object storage: Upload parts via presigned URLs
  3. 3Video API → Object storage: Complete multipart upload, verify object
  4. 4Video API → Postgres: Video row and job row in one transaction
  5. 5Transcode workers → Postgres: Claim lease, heartbeat, fenced completion
  6. 6Transcode workers → Object storage: Read raw upload, write attempt output
  7. 7Reconciler → Postgres: Find abandoned uploads and orphaned output
  • Request / response
  • Bulk data

What you need to know

0 of 2 checks done

  1. When several writes together make something visible, their order decides what a reader can see in between. A safe rule: write the things being pointed to before the thing that points to them.

    For a video: segments are pointed to by the manifest, and the manifest is pointed to by the video's status. So segments first, then manifest, then status.

  2. Think first

    Suppose a worker flipped the status to 'ready' first, then wrote the manifest and segments. What could a student see?