GPU fabric and failures
GPU training cluster
- Fit the model in one server
- Data parallel on the rails
- Checkpoints without stalls
- A straggler and a failed server
- Final test
Train a 30B model on 5 servers of 8 H100. Each stage asks more of the cluster: fit the model, spread it over the rails, save it without stalls and keep a long run alive.
Final testRail switch 3 fails for 1 h in the long run. Your final board runs again.
Stage 4 of 4 · 53 parts given · lab events only
Progress on this page stays on this device.