Skip to content
sysgeeks

Design a Video Processing Pipeline, stage 6 of 14: decide

Showing the instructor what is happening

An instructor watches the video page after uploading. Status changes perhaps four times over 20 minutes: queued, processing, ready or failed. The status lives in Postgres and is changed by workers, which are separate processes from the API servers the browser talks to.

System so far· 8 parts
123456789CLIENTInstructorbrowserSERVICEVideo APIDATABASEPostgresOBJECT STOREObject storageWORKERTranscodeworkersWORKERReconcilerEDGECDNCLIENTStudent player

Select a component to see what it is responsible for and which state it owns.

  1. 1Instructor browser → Video API: Create upload, report parts, poll status
  2. 2Instructor browser → Object storage: Upload parts via presigned URLs
  3. 3Video API → Object storage: Complete multipart upload, verify object
  4. 4Video API → Postgres: Video row and job row in one transaction
  5. 5Transcode workers → Postgres: Claim lease, heartbeat, fenced completion
  6. 6Transcode workers → Object storage: Read raw upload, write attempt output
  7. 7Reconciler → Postgres: Find abandoned uploads and orphaned output
  8. 8CDN → Object storage: Origin fetch on cache miss
  9. 9Student player → CDN: Manifest and segments
  • Request / response
  • Bulk data

What you need to know

0 of 2 checks done

  1. There are three common ways for a page to learn about changes:

    MethodHowGood for
    Pollingthe client asks every few secondsrare changes, delays of seconds acceptable
    Server-Sent Eventsthe server keeps a one-way stream openfrequent server-to-client updates
    WebSocketsa two-way connectionboth sides sending often (chat, collaboration)

    See Server push: polling, long polling, SSE, WebSockets.

  2. Work it out

    200 instructors have the upload page open, and each page polls every 5 seconds. About how many requests a second does that add?
    requests per second