Ring all-reduce time

Fabric 04

Estimate

8 GPUs must add up a 10 GB gradient so that each GPU gets the full sum. Each GPU has one 400 Gbit/s port to the switch. They use a ring all-reduce: each GPU sends 2 x 7/8 of the tensor to its neighbor. How many ms does it take?

100 µs1 ms10 ms100 ms1 s10 s100 s

Type a number or drag the scale. Each mark is 10 times the mark before it.

A full batch keeps each GPU busyArrow keys change the GPUs. Tap or Space changes the batch