Stream the checkpoints
Fabric 09Fix the board. Turn every goal green.
Goals
Not tested yet
32 H100 train a 13B model. Every 20 steps all ranks save 182 GB of weights and optimizer state, and the checkpoint store takes 10 GB/s. A GPU may fail at any time. Keep the GPUs busy 95 percent of the run and lose at most 300 s of work in a failure. One change.
11:40 Goodput 87 percent. Graph: GPU use drops to 0 for 18 s every 140 s, store write flat at 10 GB/s.
5 MB/s · logs kept on the host · disk full in 33 minutes