PCIe Switch Topologies for 8-GPU Servers: P2P, ACS, IOMMU

Why topology decides PCIe multi-GPU performance

In an 8-GPU server without NVLink across all eight cards, every collective, host-to-device copy and GPUDirect RDMA transfer travels over a PCIe tree fixed by the motherboard and switch configuration. Two servers with identical parts can behave very differently under NCCL because one keeps GPU-to-GPU traffic inside a PCIe switch while the other forces it through a CPU root complex or across sockets. The matrix printed by nvidia-smi topo -m should be read as a design document: it states where peer-to-peer (P2P) traffic physically goes.

NVIDIA lists the H100 SXM at 900 GB/s of NVLink bandwidth and the H100 NVL at 600 GB/s of NVLink, with 128 GB/s of PCIe Gen5 on both ^[1]. The L40S has no NVLink at all and lists "PCIe Gen4 x16: 64GB/s bidirectional" ^[2]; the RTX PRO 6000 Blackwell Server Edition is a PCIe Gen5 card ^[3]. On such systems PCIe is the GPU fabric, and its layout is the main performance variable left to the operator.

What the PCIe numbers actually mean

PCI-SIG defines PCIe 5.0 at 32.0 Gb/s per lane per direction of raw bandwidth, and states that a 32-lane PCIe 5.0 link provides 128.0 GB/s "in each direction simultaneously" ^[4]. PCIe 3.0 and later use 128b/130b encoding instead of 8b/10b ^[5]. The derived per-direction figures for a Gen5 x16 GPU port are:

1. Raw rate: 32 Gb/s per lane x 16 lanes = 512 Gb/s per direction = 64 GB/s per direction.
2. After 128b/130b encoding: 64 x 128/130 = ~63.0 GB/s per direction.
3. Both directions together: ~126 GB/s, which is the figure vendors round to "128 GB/s" when they quote raw bidirectional bandwidth.

Packet headers and flow control reduce delivered bandwidth further; the real figure depends on payload size and platform and must be measured.

Building blocks

Root complexes and host bridges. Each CPU socket exposes PCIe root ports behind one or more host bridges. Two sockets are joined by an SMP interconnect (UPI on Intel, xGMI on AMD, per Supermicro's GPU server root architecture brief).

PCIe switches. A switch fans upstream links out to many downstream ports. Broadcom's PEX89000 Gen5 family is offered in 144-, 104-, 88-, 72-, 48-, 32- and 24-lane configurations, supports link widths from x1 to x16, quotes "full line rate on all ports" and a cut-through latency of "less than 115 ns (x16 to x16)" ^[6]. Its quoted 1024 Gb/s (128 GB/s) raw bandwidth per x16 port is the bidirectional raw figure derived above.

Endpoints. GPUs, NICs and NVMe drives. NIC placement matters as much as GPU placement, because GPUDirect RDMA moves data directly between GPU and NIC.

Three reference layouts

Supermicro's brief describes three PCIe GPU server architectures ^[7]:

Table 1. PCIe GPU server layouts

For collective-heavy work on PCIe cards, the deciding question is which GPU pairs, and which GPU-NIC pairs, share a switch.

Reading nvidia-smi topo -m

The nvidia-smi documentation (fetched 2026-10-28; v610 is the newest changelog branch) defines the legend ^[8]:

Table 2. nvidia-smi topo -m legend

Two details are easy to miss. The newer topo -gpu and topo -nvme legends word PIX as "at most a single PCIe bridge" and PXB as "multiple PCIe bridges", so a PIX entry should be checked against the vendor block diagram rather than assumed to be one switch chip. And the matrix describes the tree statically; it does not show whether ACS redirects traffic upward. For that, the documentation provides nvidia-smi topo -p2p p (P2P PCIe capability per GPU pair) and nvidia-smi topo -p -i <id1>,<id2> (most direct PCIe path between two GPUs). The v610 changelog adds topo -gpu, topo -nic and related subcommands, plus a "DR - Disabled by regkey" item for topo -p2p.

How P2P is routed

NVIDIA's GPUDirect RDMA documentation (CUDA 13.4) states the hard requirement: the two devices "must share the same upstream PCI Express root complex." It then ranks paths: where "there are only PCIe switches on the path," the result "is optimal and yields the best performance"; a path through a single CPU/IOH "works, but yields worse performance"; and a path across a QPI/HT link "may be extremely performance-limited or even not work reliably" ^[9]. The text addresses GPU-NIC transfers, but the physical ordering applies equally to GPU-GPU P2P.

The Linux P2PDMA documentation (kernel 7.3.0) explains why the root complex is the weak point. Behind a switch, "based on the ACS settings the transaction can route entirely within the PCIe hierarchy and never reach the root port," and the kernel "will evaluate the PCIe topology and always permit P2P in these well-defined cases." Beyond that, because "the PCIe specification doesn't define the forwarding of transactions between hierarchy domains," the kernel "defaults to blocking such routing," with an allow list for known-good host bridges ^[10]. CUDA P2P enablement is decided separately by the NVIDIA driver, so nvidia-smi topo -p2p p is the authoritative answer for a given box.

In matrix terms, PIX and PXB avoid the CPU; PHB and NODE depend on its forwarding; SYS adds the socket link that NVIDIA flags as potentially unreliable.

ACS: what it is and why it breaks P2P

Access Control Services (ACS) are per-port PCIe controls for policing peer-to-peer traffic. The kernel's config_acs parameter lists seven: Source Validation, Translation Blocking, P2P Request Redirect, P2P Completion Redirect, Upstream Forwarding, P2P Egress Control and Direct Translated P2P ^[11]. With P2P Request Redirect enabled on a switch downstream port, a GPU-to-GPU request under the same switch is sent upstream to the root complex, where the IOMMU can inspect it, instead of port-to-port.

The consequence is that a PIX pair in nvidia-smi topo -m can behave like a PHB pair. NCCL's troubleshooting guide (NCCL 2.32.3) says IO virtualization can interfere with GPU Direct by "redirecting all PCI point-to-point traffic to the CPU root complex," causing "a significant performance reduction or even a hang" ^[12]. Redirected traffic from every pair under a switch then shares the switch uplink.

NCCL's bare-metal check is sudo lspci -vvv | grep ACSCtl; SrcValid+ indicates ACS might be enabled ^[12]. The flags that force traffic upstream are ReqRedir and CmpltRedir on the downstream ports above each GPU.

IOMMU and virtualization

ACS exists largely to make the IOMMU meaningful: if devices could talk directly, an IOMMU at the root complex could not isolate them, so the two settings are usually changed together.

On bare-metal Linux, NCCL 2.32.3 states that "CUDA and the NVIDIA driver stack do not support IOMMU-enabled PCIe peer-to-peer memory transfer" and that "the IOMMU must be disabled on Linux bare-metal systems to prevent silent device memory corruption." It treats dmesg messages such as iommu: Default domain type: Translated or DMAR: IOMMU enabled as signs of translated mode, to be avoided for GPU PCIe P2P ^[12]. The CUDA 13.4 GPUDirect RDMA documentation is less strict for GPU-NIC traffic: IOMMUs "must be disabled or configured for pass-through translation for GPUDirect RDMA to work" ^[9]. The relevant Linux options are iommu=pt (equivalent to iommu.passthrough=1), intel_iommu=off and iommu=off ^[11]. The NCCL page never mentions passthrough: its fix is to "avoid translated IOMMU mode" ("Disable IOMMU translation for the GPU path"), which an identity (pass-through) domain meets in principle, but the CUDA guide it cites says disabled. Validate passthrough for GPU-GPU P2P per platform with simpleP2P and nvbandwidth rather than assume it.

Virtual machines invert the requirement. NCCL's guide states that "virtual machines require ACS to function, hence disabling ACS is not an option," and that for maximum performance in VMs, "ATS needs to be enabled in network adapters" ^[12]. With Address Translation Services, a device issues requests that are already translated, and ACS has a separate Direct Translated P2P control for those requests, so a virtualized host can keep isolation for untranslated traffic while allowing an efficient NIC path. Guests also need a faithful PCIe tree: NCCL "relies on /sys to discover the PCI topology."

Disabling ACS safely

Three methods turn ACS redirection off on a single-tenant bare-metal host; the first two come from the NCCL 2.32.3 troubleshooting guide, the third from the kernel parameters documentation:

Table 3. ACS disable methods

The cost is explicit: disable_acs_redir "removes isolation between devices and may put more devices in an IOMMU group" ^[11]. On a multi-tenant host that matters: IOMMU groups are "the unit of ownership used by VFIO", and the VFIO documentation cites a bridge "allowing redirection without reaching the IOMMU" as a cause of reduced isolation ^[13]; once ACS is off, GPUs under one switch typically can no longer go to different guests, and each can reach its peers unseen by the IOMMU. The working rule: disable ACS only on single-tenant bare-metal hosts; keep it on, with ATS on NICs, for hosts that serve VMs.

A setpci fix that is not reapplied after reboot produces nodes that pass acceptance and later regress, so the persistence mechanism belongs in the node image and the check in the boot-time health gate, next to the checks in PCIe link health troubleshooting.

NCCL on PCIe-only boxes

NCCL selects the P2P transport when CUDA reports that GPUs can access each other's memory ^[12]. Two environment variables in NCCL 2.32.3 override that decision ^[14]:

• NCCL_P2P_DISABLE=1 disables "direct GPU-to-GPU (P2P) communication" entirely.
• NCCL_P2P_LEVEL sets "the maximum distance between GPUs where NCCL will use the P2P transport," using the same vocabulary as the topology matrix: LOC (never), NVL, PIX, PXB, PHB and SYS. If unset, NCCL selects a value from the architecture; legacy integers (LOC=0 to SYS=4) are discouraged.

The GPU-NIC equivalent, NCCL_NET_GDR_LEVEL, caps GPU-NIC distance for GPUDirect RDMA with the same LOC/PIX/PXB/PHB/SYS values. NCCL_NET_GDR_READ defaults to 1 on NVLink-based platforms and 0 otherwise, and the documentation notes that reading directly from GPU memory "is known to be slightly slower" on some platforms, such as PCI-E ^[14].

When to force P2P off

The NCCL guide uses these variables diagnostically. Three situations justify a persistent override:

1. Hangs or corruption on PHB/SYS pairs that persist after ACS and IOMMU are corrected: cap with NCCL_P2P_LEVEL=PXB so P2P is used only where no CPU forwarding is involved.
2. VM guests where ACS must stay on: P2P through the redirected path may be slower than NCCL's non-P2P path through host memory; measure both.
3. Mixed designs where some GPU pairs are PIX and others SYS: a level cap keeps the fast pairs on P2P without forcing the slow ones.

Overrides should be recorded per node type, together with the XML written by NCCL_TOPO_DUMP_FILE ^[14].

Design patterns for 8 GPUs

The design question is which of three flows share which links: GPU-GPU collectives, GPU-NIC traffic, and host-to-GPU loading.

Lane budget (illustrative)

Taking the 144-lane PEX89000 configuration as a reference and x16 for every GPU, NIC and uplink:

• Eight GPUs on one switch: 8 x 16 = 128 lanes for GPUs, plus one x16 uplink = 144 lanes. Every GPU pair is PIX, but no lanes remain for NICs and host traffic is oversubscribed 8:1.
• Four GPUs per switch, two switches: 4 x 16 = 64 GPU lanes, plus 2 x 16 = 32 NIC lanes, plus 2 x 16 = 32 uplink lanes = 128 lanes per switch. Host oversubscription is 64:32 = 2:1, and every GPU has a PIX NIC shared with one other GPU.

These are derived counts, not a specific product; real boards also spend lanes on NVMe and management.

Uplink oversubscription arithmetic (illustrative)

For Supermicro's single-root description (each switch on 2 x16 uplinks, up to five GPUs), with ~63.0 GB/s per direction per x16 port after encoding:

1. Uplink capacity per direction: 2 x 63.0 = 126 GB/s.
2. Demand if five GPUs read from host memory at once: 5 x 63.0 = 315 GB/s.
3. Ratio: 315 / 126 = 2.5:1, so each GPU gets at most ~25.2 GB/s from the host (126 / 5), before CPU memory and root-port limits.

An illustrative 400 Gb/s NIC port carries 400 / 8 = 50 GB/s per direction. Under the same switch as its GPUs (PIX), GPUDirect RDMA traffic never touches the uplink; under the other socket (SYS), those 50 GB/s cross both uplinks and the socket interconnect, competing with host traffic.

Ring placement

An 8-GPU ring spanning switches on two sockets crosses the inter-switch cut once in each direction. With P2P across SYS, ring bandwidth is bounded by the slowest of the GPU port, the uplink and cross-socket P2P forwarding, which NVIDIA warns may be "extremely performance-limited." With P2P capped at PXB, the crossing goes through host memory instead. Neither outcome is predictable from the matrix alone. Rank-to-NIC pairing is covered in CPU/NIC affinity and rank binding.

Table 4. Pattern checklist

Validation runbook

Table 5. Validation steps

Steps 2 to 5 follow NCCL's troubleshooting guide, which recommends nvbandwidth over the CUDA sample for bandwidth ^[12][15]. GPU-NIC paths need their own test, described in GPUDirect RDMA qualification, and the results belong in the node record defined by the GPU cluster acceptance testing runbook.

The NCCL guidance quoted here is from the NCCL 2.32.3 documentation (v2.32.3-1, released 2026-09-17, still the latest release on 2026-10-28); it is version-specific and should be re-read for the deployed release. Bandwidth figures here are derived from line rates; only measured results belong in a capacity plan.

GPU Smith can review a PCIe server's block diagram and topology output against the intended workload before purchase or acceptance.