Ring all-reduce time
Difficulty 3 of 5
8 GPUs must add up a 10 GB gradient so that each GPU gets the full sum. Each GPU has one 400 Gbit/s port to the switch. They use a ring all-reduce: each GPU sends 2 x 7/8 of the tensor to its neighbor. How many ms does it take?
The question
Answer with one number in ms.
Goals and limits
- Completion time
- under 800 ms
- Monthly cost
- under 100,000 USD/mo
Concepts
How players did
Player results load from the server.