Distributed compute
Distributed compute splits one large job across many machines or GPUs, so that it ends sooner. One L4 GPU host costs 520 USD/mo, so use each GPU well before you add more. A batch raises the work of each GPU, and more GPUs divide the time.
See it on the board
The same traffic runs on two boards. The simulator measured both when this page was built.
| Measure | Goal | One GPU with batch size 1 | Two GPUs with batch size 64 |
|---|---|---|---|
| Completion time | under 86,400 s | 90,000 s Fail | 74,100.25 s Pass |
| Lost writes | none | 0 Pass | 0 Pass |
| Monthly cost | 584 USD/mo | 1,104 USD/mo | |
| Error rate | 0% | 0% | |
| Level rules | none broken | Board is not complete Fail | Pass |
Key numbers
| Number | Value |
|---|---|
| Price of one L4 GPU host | 520 USD/mo |
| Price of one H100 GPU host | 1,900 USD/mo |
| GPU memory, one L4 host | 24 GB |
| GPU memory, one H100 host | 80 GB |
2 GPUs · batch 64 · job 1.5 h · 2.1 USD
How it works
Some jobs are too large for one machine: embed 100 million items, train a model, render a film. Distributed compute splits the work into many pieces and runs them on many machines at the same time.
The easy case has independent pieces. Each item can go to any worker, and its result does not depend on other items. A queue hands out the pieces, and the job time falls almost in proportion to the number of workers. To embed a catalog is such a job.
Before you add machines, make each one efficient. A GPU that handles one item at a time is idle for most of each small step. A batch lets one pass handle many items, which can raise throughput several times on the same hardware. A batch that is too large fills GPU memory and becomes slow again.
Harder cases need the workers to talk. Training a large model splits the data or the model across GPUs, and they exchange results at each step. The network between them then becomes the limit, and each added GPU gives less.
Plan for failure. With many machines, one of them will fail during a long job. Save progress in small pieces, so that a restart repeats only a little work.
In the levels you batch the work on each GPU, then add GPUs until the job meets its deadline and its budget.