Fit the model

Level 36 of 41

Choose

A decode step reads 140 GB of weights. Each H100 holds 80 GB. Pick the design that finishes within 30 ms. The times include memory reads and data transfers.

Goals

Not tested yet

99 of 100 requests finish faster than this.What the servers and services cost each month.
Choose a design
Tensor parallel
Pipeline
CPU offload