GPU SmithCOMPUTE ENGINEERING

megatron tensor parallel within node

1 article

Mapping TP, PP and DP to NVLink and InfiniBand Domains

Per-step traffic formulas for tensor, pipeline, context and data parallelism, where each belongs on NVLink or InfiniBand, and a 405B, 1,024-GPU example.

October 28, 2026• 17 min read
tensor parallel pipeline parallel data parallel nvlink infiniband placementmegatron tensor parallel within node3d parallelism communication volume