How to Size an LLM Training Cluster: A Practical Framework
Sizing a training cluster wrong means either paying for idle GPUs or running out of memory mid-training. This framework walks through the variables that actually determine how many accelerators you need.
The variables that matter
Cluster size is driven by a handful of inputs, not guesswork:
- Model parameters — the number of weights in your model.
- Precision — BF16, FP8, or mixed precision changes the memory footprint dramatically.
- Optimiser state — AdamW and its variants store two states per parameter.
- Gradient checkpointing — trades compute for memory.
- Batch size and sequence length — activation memory scales with both.
The memory math
A reliable rule of thumb for full fine-tuning: budget roughly 6–8× the model's weight size in total memory, accounting for weights, gradients, and optimiser states. For a 70B-parameter model in BF16, weights alone are ~140 GB, so total memory lands around 840 GB to 1.1 TB.
A worked example: a 70B LLM
| Component | Memory |
|---|---|
| Model weights (70B params · BF16) | ~140 GB |
| AdamW optimiser states | ~280 GB |
| Gradients | ~140 GB |
| Subtotal (no activations) | ~560 GB |
| With activations and batch | ~1 TB total |
| GPU count | 8 × MI300X (1,536 GB aggregate) |
That leaves comfortable headroom. The same 70B model would require roughly 16 × H100 (80 GB) just to match the memory — a reminder that memory capacity, not raw FLOPS, is often the binding constraint.
Network and storage
Multi-node training depends on the interconnect. InfiniBand or RoCE at 400 Gbps is the practical standard for distributed training, and your storage layer must keep pace with dataset loading and checkpoint writes. A cluster is only as fast as its slowest component — a bottlenecked file system will idle expensive GPUs.
Budget and timeline
Finally, factor in availability. NVIDIA lead times can derail a project before it starts. AMD Instinct capacity is often deliverable on meaningfully shorter timelines, which can be worth more than a few percentage points of peak throughput when you are racing a deadline.
The bottom line: start with the memory math, then validate with real benchmarks. Leave 20–30% headroom for larger batches and future fine-tuning runs. Do not let networking or storage become the bottleneck.