GPU SmithCOMPUTE ENGINEERING

multi node vllm pipeline parallelism ray

1 article

Tensor vs Pipeline Parallelism for Inference: NVLink vs PCIe

How to split an LLM across GPUs for inference: per-token all-reduce traffic for tensor parallelism, pipeline trade-offs, NVLink vs PCIe worked examples.

October 28, 2026• 15 min read
tensor parallel vs pipeline parallel inferencevllm tensor_parallel_size pipeline_parallel_sizetensor parallelism pcie vs nvlink all-reduce