A week of training

  • Survive
  • Fabric 10
  • Datacenter

Difficulty 4 of 5

16 H100 train a 13B model for a week; a third server waits idle. A GPU, a rail switch and a whole server will fail. Set the checkpoint plan and the spare pool before the run: the job must be productive 96.5 percent of the week and lose at most 40 min of work in a failure.

Play level

Data centerHostHostHostTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPTCPGPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)GPU (H100)NVLink sw…NVLink sw…NVLink sw…InfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 nsInfiniBan…200 ns
The board at the start of the level
A backup and its restoreArrow keys change the backup interval. Tap to change the restore rate.

Backup every 60 minutes · RPO 60 minutes · RTO 11 hours at 100 Mbps

Goals and limits

Data lost (RPO)
under 2,400 s
Availability
at least 90%
Monthly cost
under 1,000,000 USD/mo
Waves
3
Time within SLO
at least 96.5%

Concepts

How players did

Player results load from the server.

All levels