Design a Distributed Job Queue, stage 1 of 9: model
What the numbers say
Use 86,400 seconds in a day, and about 2 KB per job.
System so far· 4 parts
Select a component to see what it is responsible for and which state it owns.
- 1Workers → Redis queues: Lease jobs
- 2Workers → Databases and services: Do the work
What you need to know
0 of 3 checks done
A day has 86,400 seconds. Divide a daily total by 86,400 to get the average per second, then compare it with the peak to see how spiky the traffic is.
Work it out
1.4 billion jobs a day. About how many jobs a second is that on average?A queue holds the difference between what arrives and what is finished. If workers keep up, the queue stays near empty. If they slow down, the backlog grows by (arrival rate − completion rate) every second, for as long as that lasts.
So the question for any buffer is: how long a bad period can it hold, and what does that cost?
Work it out
Workers stop completely for 10 minutes at the 33,000-a-second peak. Jobs are about 2 KB. About how many gigabytes of jobs pile up?Check
Where is it cheapest to hold a backlog that might reach 40 GB a few times a year?