Drain the straggler

  • Fix
  • Fabric 07
  • Rack

Difficulty 3 of 5

Eight H100 train an 8B model with data parallel; a ninth GPU is a hot spare on the same switch. Each step ends with a ring all-reduce, so every GPU waits for the slowest. A step took 1.55 s last week. Bring the median step back under 1.75 s with one change.

03:10 Step time 2.15 s, was 1.55 s. Graph: kernel time per GPU; gpu 5 is 1.6 times the others.

Play level

Zone RackTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)InfiniBand…200 ns
The board at the start of the level
A full batch keeps each GPU busyArrow keys change the GPUs. Tap or Space changes the batch

2 GPUs · batch 64 · job 1.5 h · 2.1 USD

Goals and limits

p50 latency
under 1.8 s
Monthly cost
under 100,000 USD/mo
Edit limit
1

Concepts

How players did

Player results load from the server.

All levels