Stream the checkpoints
Difficulty 4 of 5
32 H100 train a 13B model. Every 20 steps all ranks save 182 GB of weights and optimizer state, and the checkpoint store takes 10 GB/s. A GPU may fail at any time. Keep the GPUs busy 95 percent of the run and lose at most 300 s of work in a failure. One change.
11:40 Goodput 87 percent. Graph: GPU use drops to 0 for 18 s every 140 s, store write flat at 10 GB/s.
5 MB/s · logs kept on the host · disk full in 33 minutes
Goals and limits
- Availability
- at least 95%
- Data lost (RPO)
- under 300 s
- Monthly cost
- under 1,000,000 USD/mo
- Edit limit
- 1
Concepts
How players did
Player results load from the server.