Drain the straggler

Fabric 07
Zone RackTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)InfiniBand…200 ns

Fix the board. Turn every goal green.

Goals

Not tested yet

Half of the requests finish faster than this.What the servers and services cost each month.

Eight H100 train an 8B model with data parallel; a ninth GPU is a hot spare on the same switch. Each step ends with a ring all-reduce, so every GPU waits for the slowest. A step took 1.55 s last week. Bring the median step back under 1.75 s with one change.

03:10 Step time 2.15 s, was 1.55 s. Graph: kernel time per GPU; gpu 5 is 1.6 times the others.

A full batch keeps each GPU busyArrow keys change the GPUs. Tap or Space changes the batch

2 GPUs · batch 64 · job 1.5 h · 2.1 USD