distributed training
2 articles
NCCL Algorithms and Protocols: Selection and Debugging
A 2026 guide to NCCL algorithms and protocols, covering Ring, Tree, LL, LL128 and NVLS, versioned controls, tuning logs, bandwidth formulas and reversible diagnostic experiments.
October 12, 2026• 22 min read
nccl algorithms and protocolsnccl ring vs treenccl ll128
LLM Training Memory Calculator for ZeRO and FSDP
A 2026 planning model for per-rank LLM training memory, ZeRO and FSDP sharding, GPU count, CPU or NVMe offload, topology rounding, and measured validation.
September 18, 2026• 21 min read
llm training memory calculatorzero memory calculatorfsdp memory calculator