Design a Distributed Job Queue, stage 7 of 9: break it
Write the worker loop
Write the loop each worker runs for one job type. The queue supports leases, acknowledgements, delayed retries and a dead-letter queue.
System so far· 7 parts
Select a component to see what it is responsible for and which state it owns.
- 1Web servers → Enqueue gateway: Enqueue job
- 2Enqueue gateway → Kafka: Append to topic
- 3Relay → Kafka: Read topics
- 4Relay → Redis queues: Push at a controlled rate
- 5Workers → Redis queues: Lease jobs
- 6Workers → Databases and services: Do the work
What you need to know
0 of 3 checks done
A worker loop handles one job at a time, and each job ends in exactly one of these ways:
Outcome When Queue call Done the handler succeeded ackRetry later it failed, attempts remain retryLaterwith a delayDead letter it failed too many times deadLetterExpired it's too old to matter ackwithout runningCheck
Where should the acknowledgement go?Jitter spreads retries out. If 1,000 jobs fail at the same moment and all wait exactly 4 seconds, they all retry at the same moment, too. "Equal jitter" waits half the backoff plus a random amount up to the other half:
base = min(1000 × 2^attempts, cap) delay = base / 2 + random() × base / 2Work it out
With equal jitter, attempt 3 has base = 1000 × 2³ ms. What's the longest delay it can get, in seconds?Think first
What should the loop do when lease() returns no job?