GPU SmithCOMPUTE ENGINEERING

cpu offload

1 article

LLM Training Memory Calculator for ZeRO and FSDP

LLM Training Memory Calculator for ZeRO and FSDP

A 2026 planning model for per-rank LLM training memory, ZeRO and FSDP sharding, GPU count, CPU or NVMe offload, topology rounding, and measured validation.

9/19/2026• 21 min read
llm training memory calculatorzero memory calculatorfsdp memory calculator