Back to Articles|Published on November 20, 2026|28 min read

GPUSmith Article

Multi GPU PCIe Peer Access: Purchase Evidence Matrix

Purchase evidence matrix for multi GPU PCIe peer access in 2026: separates vendor support, runtime capability, observed transport and workload benefit, with ledger fields, IOMMU/ACS notes and dispositions.

By GPU Smith

Multi GPU PCIe Peer Access: Purchase Evidence Matrix

Summary

  1. 01A peer-access claim needs four kinds of evidence: vendor support for the exact configuration, directional runtime capability, observed transport and correctness, and an application result that matters to the purchase.
  2. 02Record one acceptance-ledger row per ordered GPU pair. Run transfers in both directions and under concurrent load, compare with a host-staged control, and save the transport shown in the engine logs.
  3. 03Keep an exact configuration's support status unknown when no vendor statement was located. Peer access demonstrated with a patched consumer driver belongs in the observed-community-only disposition.
  4. 04The documented IOMMU and ACS requirements differ between bare-metal Linux and virtual machines. Record the isolation posture the vendor tested and who approved it.
  5. 05Use repeated, matched workload runs to see whether peer access changes latency or qualified requests per service group. Independent replicas may meet the service objective without peer access.
Inside this article
  1. 01Executive Summary
  2. 02Introduction and Background
  3. 03Path Terms and the Four Claims
  4. 04Hardware and Platform Evidence Matrix
  5. 05Software Paths and Fallbacks
  6. 06Acceptance Evidence and Vendor Schedule
  7. 07Data Analysis and Evidence
  8. 08Implications and Future Directions
  9. 09Frequently Asked Questions (FAQs)
  10. 10Conclusion

Executive Summary

A quoted multi-GPU PCIe server that "supports peer access" bundles four separate claims: advertised hardware eligibility, runtime directional capability, observed transport and correctness, and useful application performance. NVIDIA's CUDA programming guide states that, depending on the system properties, specifically the PCIe and/or NVLink topology, devices can address each other's memory [1]. Access must be enabled explicitly between two devices, and the runtime reference states that the access granted by an enable call is unidirectional [2]. Once enabled, copies no longer need to be staged through the host [1]. As of November 2026 the guide's footer reads "Last updated on Oct 31, 2026" [1], so buyers should record the revision they actually read.

The vendor evidence is thinner than most quotations imply. NVIDIA's RTX PRO 6000 Blackwell Server Edition page states PCIe Gen 5 support [3], and a Lenovo platform guide says no NVLink bridge is applicable to that card [4], but the pages read for this report did not state a directional P2P guarantee for any specific server bill of materials. Those cells stay unknown in the matrix below. For GeForce cards, an NVIDIA developer forum post relays engineering feedback that peer to peer is not supported on the RTX 4090 [5] (a February 2023 community-hosted statement, not a current support certificate), and llama.cpp's guide says P2P requires driver support that is usually restricted to workstation and datacenter GPUs [6].

Platform settings decide whether a capable pair stays capable. NVIDIA documents that the IOMMU must be disabled on bare-metal Linux for PCIe P2P [1], and that ACS on PCI switches needs to be disabled [7], while virtual machines require ACS. Those requirements pull in different directions, so platform and security owners must sign the isolation posture; this report does not recommend disabling isolation globally.

Measured benefit is workload specific. A 2026 preprint on an eight-GPU RTX 5090 server reports one-way bandwidth rising from 31.5 GB/s to 56 GB/s once P2P was enabled with a modified driver [8], while a self-reported vLLM forum experiment on two RTX 5090 cards describes itself as an exploratory point-estimate matrix, not a multi-rep confidence interval [9]. The preprint's authors state that they used a modified version of the open-source NVIDIA driver and disabled IOMMU and PCIe ACS, among other platform changes, to enable P2P, so the gain depended on those changes. Modifying drivers or deploying GeForce cards in a data center may carry licence and warranty implications, so confirm them with the vendor and legal counsel before buying. A faster copy path is a capability result, not a latency, throughput or purchase-count result.

The recommended posture uses five dispositions: supported-and-proven, supported-but-unproven, observed-community-only, unsupported, and unknown. Each needs a named support owner, dated evidence, a host-staged control run, and an engine log showing which transport actually ran. If a service-level objective is met by independent replicas, the purchase may not need peer access at all.

31.5 GB/sHost-staged one-way bandwidth on the eight-GPU RTX 5090 test system
56 GB/sOne-way bandwidth with peer access enabled on the same modified-driver RTX 5090 test system
23.4%CoMoE's reported share of the NVLink-capable A800 hardware cost in its mixture-of-experts comparison
20%Upper end of the distributed LLM inference overhead reported by TokenWeave

Introduction and Background

Enterprise AI platform leads, integrators and procurement engineers who buy two to eight PCIe GPUs for private inference face a narrow but expensive question: does the quoted configuration support the direct GPU-to-GPU path that the tensor-parallel or pipeline-parallel workload needs, and if so, does that path change how many GPUs must be purchased? The answer depends on the exact GPU board, server, PCIe switch or riser, firmware, driver, operating system, virtualization layer and serving engine. No generation-wide yes or no cell survives that list.

GPU Smith is an independent private AI infrastructure engineering consultancy that advises on GPU procurement, workload sizing, networking, inference optimization, power and cooling, and air-gapped deployments. It is not a chip manufacturer, server vendor or cloud provider, and it sells none of the products discussed here. No hardware tests were performed in preparing this report. Every measurement quoted below belongs to a named third-party source with its own hardware assumptions.

The reusable pattern in hardware-selection literature is explicit assumptions plus an explanation of which architectural constraint changes the conclusion. Tim Dettmers' widely read hardware guide is a dated decision aid: its version history lists 2023-01-30 as its latest entry [10]. The NeurIPS 2024 SpecExec paper cites it as one basis for defining a popular consumer hardware configuration (Source: papers.neurips.cc), and a January 2024 CNAS policy report cites it for the price comparison between gaming and data center GPUs [11]. These are citations of a hardware-selection reference, not endorsements of a measured P2P result, and they are not current support certification. The guide's general statement that PCIe lanes are seldom a bottleneck [10] is the kind of dated generalization a 2026 purchase should replace with a configuration-specific test.

Server-class and consumer-class parts sit under different support statements. NVIDIA's RTX PRO reference architecture describes systems in two, four and eight GPU configurations [12], while an October 2026 preprint on commodity-GPU mixture-of-experts inference states that consumer GPUs connect via standard PCIe buses and lack P2P support, and designs around host-mediated transfers [13]. The buyer's job is to attach the correct statement to the correct line item. The sections below define path terms, assemble a vendor-evidence matrix, map engines to fallbacks, set an acceptance schedule, and quantify what published numbers can and cannot support.

Path Terms and the Four Claims

Path vocabulary

Procurement documents mix terms that describe different mechanisms. These definitions keep a contract from accepting evidence for one path as evidence for another.

  • GPU-to-GPU peer copy and load/store. One device reads or writes another device's memory over PCIe or NVLink; enabled peers share a unified virtual address space.
  • GPU-to-NIC RDMA. A different path between a GPU and a network adapter. NVIDIA's GPUDirect RDMA documentation requires that the two devices share the same upstream PCI Express root complex [14]. A passing GPU-NIC test says nothing about GPU-to-GPU access.
  • NVLink. A direct GPU interconnect, shown in the nvidia-smi legend as a bonded set of NVLinks. Some PCIe cards use an optional bridge, covered in Table 1.
  • Host staging. Data moves from one GPU into host memory and out to the other. Direct peer copies avoid this staging, which makes it the control path a buyer should always measure.
  • CUDA IPC and virtual memory management. Processes exchange allocation handles, and the virtual memory management API requires access control to be selected explicitly with cuMemSetAccess.
  • Engine collectives. NCCL, a custom all-reduce kernel or an engine's own copy path. NCCL favors peer-to-peer transport when CUDA reports that GPUs can access each other's memory [7], but other engines make their own decisions.

The four claims

The first claim is advertised eligibility: a datasheet, quote or signed support letter says the configuration qualifies. The second is runtime directional capability: cudaDeviceCanAccessPeer queries whether a device may directly access a peer device [2]. Because the enable call grants access that is unidirectional [2], the capability ledger needs one row per ordered pair. The runtime reference also notes device-wide and system-wide limitations per system configuration [2], and the programming guide's per-device peer-connection limit on non-NVSwitch systems is one more item to confirm for eight-GPU boxes.

The third claim is observed transport and correctness. NCCL's logging documentation shows peer transport as via P2P/CUMEM [15] and shared-memory staging as via SHM/direct/direct [15]. vLLM's utilities state that a real P2P access must be performed because the framework-level capability query can report availability when access is not actually possible [16]. A positive query is an input to acceptance, not the acceptance itself. The fourth claim is useful application performance: repeated matched runs of the actual workload show a latency or capacity change that affects the number of GPUs or groups purchased.

What does not establish support

  • An x16 slot. The RTX PRO 6000 Blackwell Server Edition datasheet lists a PCI Express 5.0 x16 graphics bus [17]; the same text is silent on peer access.
  • A topology listing. The legend defines PHB as a connection traversing PCIe and a host bridge, typically the CPU [18]. A community guide notes that PHB is not a dead end for P2P [19], so the label alone decides nothing either way.
  • A status matrix entry. nvidia-smi topo -p2p shows the P2P status between all GPUs, given a capability [18]. A forum thread lists a status code reading "CNS = Chipset not supported" [20], which shows the matrix can report an unsupported platform even where each card is individually capable. The manual page read for this report does not define every status code, so contracts should cite the driver release notes for the exact version.
  • Aggregate VRAM. TensorRT-LLM describes tensor parallelism as splitting the model weights across multiple GPUs [21]. Capacity is shared as shards, and the sources reviewed do not describe pooled memory as a feature of peer access.
Figure 01
Direct peer access and host staging
Direct peer access
  • One GPU reads or writes another GPU's memory over PCIe or NVLink.
  • NCCL documents the log marker via P2P/CUMEM for peer transport.
Host staging
  • Data passes through host memory between GPUs. Measure this path as the control.
  • NCCL documents the log marker via SHM/direct/direct for shared-memory staging.

Where evidence is missing, the right entry is unknown, and the right fallback is a measured host-staged configuration or independent replicas, accepted on their own numbers.

Hardware and Platform Evidence Matrix

Table 1 below records what vendor pages say about interconnect and peer access for representative PCIe GPUs as of November 2026. Where a page does not address P2P over PCIe, the cell reads unknown, meaning no statement was located, not that support is doubtful, and the buyer must obtain a statement for the exact server. NVIDIA's CUDA guide treats device-to-device memory access as dependent on system properties, specifically the PCIe and/or NVLink topology, so PCIe P2P on data-center and workstation boards is a topology- and platform-dependent question rather than one answered by the GPU model alone.

ConfigurationInterconnect stated by the vendor pagePeer-access statement locatedDisposition for a quote
RTX PRO 6000 Blackwell Server EditionPCIe Gen 5 [3]; a Lenovo guide says no NVLink bridge is applicable [4]None found on the product page or datasheet textUnknown until the OEM states it for the exact server
RTX PRO 6000 Blackwell Workstation EditionPCIe Gen 5 [22]; 96 GB GDDR7 with ECC on the family page [23]None foundUnknown
RTX 6000 Ada Generation48 GB GDDR6 with ECC [24]None found on the pageUnknown
L40SPCIe Gen4 x16 at 64 GB/s, labelled bidirectional [25]None foundUnknown
H200 NVL2- or 4-way NVLink bridge [26]; page marks specifications preliminary [26]NVLink bridge is the stated GPU-to-GPU pathBridged pairs: supported-but-unproven; PCIe-only pairs unknown
H100 NVL (PCIe)PCIe-based card with NVLink bridge [27]NVLink bridge is the stated GPU-to-GPU pathBridged pairs: supported-but-unproven
A100 PCIeNVLink bridge for up to two GPUs [28]Bridge for pairs onlyPairs: supported-but-unproven; other pairs unknown
GeForce RTX 4090Not stated on a vendor product page read for this reportA February 2023 forum post relays Engineering feedback that peer to peer is not supported on the 4090 [5]Unsupported per that dated statement; recheck the current driver release

Three cautions apply. Product pages are summaries, and an acceptance decision needs the server vendor's statement. The H200 page labels its own table preliminary [26], so a quote relying on it should name the revision. And one forum participant observes that NVIDIA does not publicly document the internal workings of the peer-capability API call [29]. That is a user's characterization, not a vendor statement, but it matches the gap seen here: no per-SKU support table was located, so the buyer needs the statement in writing for the quoted bill of materials.

Server and platform pages

Platform documentation supplies topology but not support. NVIDIA's reference architecture recommends a balanced PCIe topology with one Gen5 x16 link per GPU. It states that a PCIe switch may not be needed for low-cost inference servers [12], while an NVIDIA developer blog calls PCIe switches critical for maximizing inter-GPU communication bandwidth [30]. The two statements are compatible only once the buyer states which workload the server is for.

OEM pages describe layout rather than a guarantee. A Supermicro brief explains that a dual-root setup connects each CPU to several GPUs through a PLX switch [31], and lists single-root use when peer-to-peer communication (GPU to GPU) performance is not critical [31]. A 2026 Supermicro datasheet lists up to eight RTX PRO 6000 Blackwell GPUs [32]. Dell's PowerEdge XE7745 page lists eight PCIe Gen 5 x16 double-wide slots at up to 600 W [33], and ASUS describes built-in PCIe Gen 6 switches on an RTX PRO server model [34]. In the page text read for this report none of these OEM pages stated a directional GPU-to-GPU guarantee for a named GPU, and NVIDIA's datasheet says only that the GPU is available in NVIDIA-Certified servers [17]. A certification listing is not a P2P statement.

The practical consequence is a request, not an inference: ask the integrator to certify in the quote that each ordered GPU pair supports direct access on the delivered firmware, driver and operating system, and to name who owns the fix if it does not.

Software Paths and Fallbacks

Engines decide what the hardware capability is used for. Table 2 below consolidates documented behavior for engines commonly used in private inference. It is a test-planning aid, not a performance ranking.

Engine and modeParallelism and path documentedFallback or control documentedWhat to log for acceptance
vLLM, tensor parallelDistributed inference is probably unnecessary if the model fits on a single GPU [35]; for nodes without NVLink the docs name L40S as an example where pipeline parallelism is preferred [35]Custom all-reduce can be disabled to fall back to NCCL [36]; the source disables it when the platform lacks GPU P2P capability or the P2P test failed [37]Chosen all-reduce implementation, NCCL transport lines, vLLM and driver versions
vLLM, data parallelModel weights are replicated across separate instances to process independent batches of requests [38]No peer path needed between replicasQualified requests per second per replica
SGLangData-parallel replica count is configurable [39]Custom all-reduce can be disabled to fall back to NCCL [39]Selected all-reduce, NCCL log lines
llama.cppDefault split mode is layer, pipelined [40]; a tensor mode is marked experimental [40]An environment variable enables peer-to-peer access between GPUs [41]; the build guide documents a caution for some motherboard and BIOS settings, naming IOMMU as an example [41]Whether the P2P variable was set, split mode, build flags
TensorRT-LLMTensor parallelism splits weights across GPUs [21]A PCIe-targeted all-reduce optimization exists [42], with automatic fallback when no benefit is expected [42]Strategy selected at run time, message sizes

Several details change how a buyer should read the table. vLLM's environment-variable documentation says its P2P check assumes drivers can report P2P status correctly [43], so a driver that over-reports capability can still flow into engine selection. vLLM also documents a FlashInfer-based backend that targets small tensor-parallel all-reduces on a single PCIe-only node [44], so "PCIe-only" and "no optimized path" are not synonyms.

llama.cpp's multi-GPU guide summarizes the trade-off as pipeline-parallel maximizing batch throughput while tensor-parallel minimizes latency [6]. One community guide states that setting NCCL_P2P_DISABLE with llama.cpp's layer split does nothing [19], a caution against assuming that a library-level switch governs every engine. NVIDIA's NCCL documentation describes NCCL_P2P_DISABLE as disabling direct GPU-to-GPU communication [45] and warns that its tuning variables should generally not be set for production code [45]. The accepted configuration should therefore list which variables, if any, are set, and require a retest when they change.

The decision for each cell concerns the fallback, not the headline. If a custom kernel disables itself, the engine falls back to NCCL, and NCCL can in turn use shared memory when P2P is unavailable. Each hop is legitimate behavior with different throughput, so the acceptance log must show which one ran.

Acceptance Evidence and Vendor Schedule

The directional device-pair ledger

The acceptance artifact is a ledger with one row per ordered GPU pair: twelve rows for four GPUs and fifty-six for eight. Each row should carry:

  • Hardware identity. Exact GPU SKU and board revision, serial or UUID, PCI bus/device/function address, server model, riser or backplane, and PCIe switch model if any.
  • Firmware and platform. BIOS and BMC versions, IOMMU state, ACS state on each bridge, Above 4G decoding and BAR sizing, NUMA mapping, and any hypervisor.
  • Software provenance. Operating system, kernel, driver version and kernel-module provenance (stock, vendor-supplied or locally modified), CUDA, NCCL and engine versions, and container image digest.
  • Runtime query result. The cudaDeviceCanAccessPeer result in the row's direction, link attributes such as the performance rank, and the date.
  • Observed transport. The transport string from engine and NCCL logs for the workload actually used.
  • Correctness result. Integrity checks on transferred data and on end-to-end model output.
  • Vendor statement. Reference to the signed support letter, with scope, date and owner.

The ledger separates the query from the observation. A community maintainer frames the question as three independent questions stacked on top of each other [19], which maps to query, driver or kernel-module path, and engine behavior.

Acceptance tests

NVIDIA's nvbandwidth is a tool for bandwidth measurements on NVIDIA GPUs [46]. It implements copy-engine copies, which use memcpy APIs, and streaming-multiprocessor copies, which use kernels [46]. Its bidirectional tests drive traffic in both directions concurrently [46], and a pair option restricts device-to-device tests to one explicitly specified pair of GPUs [46]. The README lists a default memcpy buffer of 512 MiB [47]: note the binary MiB buffer unit against decimal GB/s in output, and fix both in the acceptance script. A pass establishes that pair on that system; it does not certify every board.

For collectives, the nccl-tests documentation defines bus bandwidth so that it can be compared with the hardware peak bandwidth independently of the number of ranks used [48]. Report algorithm bandwidth and bus bandwidth separately and compare each against the host-staged control.

The recommended sequence, none of which is an installation procedure:

  1. Pair query in both directions. Record each ordered pair's result with driver and CUDA versions.
  2. Read, write and integrity transfers. Use the allocation types the application uses, in buffer sizes that bracket its message sizes.
  3. Bidirectional and concurrent transfers. Run both directions at once, then every pair the application uses simultaneously, since contention is where single-pair tests mislead.
  4. Host-staged control. Repeat with the peer path disabled by the documented means for the layer under test, and archive both runs.
  5. Engine transport capture. Start the real engine with logging that prints transport per channel, and save the lines.
  6. End-to-end correctness. Compare model outputs on a fixed prompt set with a reference configuration.

A community guide advises running a bandwidth test on any new multi-GPU layout before trusting the topology matrix [19], and states that a passing transfer test is necessary but not sufficient [19]. A pass is one row of evidence.

Isolation, IOMMU and ACS

The documented requirements are not uniform, which is why the security owner must sign. NVIDIA's CUDA guide states that the IOMMU must be disabled when running Linux on a bare metal system [1]. For virtual machines the same guide reverses the advice: the IOMMU should be enabled and the VFIO driver used [1]. NCCL's troubleshooting page says that if PCI switches have ACS enabled it needs to be disabled [7], and that virtual machines require ACS, so disabling it there is not an option [7]. NVIDIA's virtual GPU documentation states that P2P transfers over PCIe are not supported [49] on that feature page.

The Linux kernel documentation explains the mechanism: with PCIe switches, ACS settings determine whether a transaction can route entirely within the PCIe hierarchy [50], and one reaching the host bridge might have to hairpin back out the same root port [50]. Forcing ACS redirect off removes isolation between devices and may put more devices in an IOMMU group [51].

The procurement reading is conservative: the schedule should state the isolation posture the vendor tested, record who approved it, and avoid a blanket instruction to turn isolation off. Shared multi-tenant or virtualized designs may need a different server or independent replicas rather than a peer path.

Support ownership and dispositions

Stock behavior and community modification must stay in separate rows. One community guide states that enabling P2P on consumer cards requires a patched kernel module [19], and a vLLM forum experiment on two consumer cards used a patched open NVIDIA kernel module so that BAR1 GPU P2P was available [9]. These report what skilled users achieved on their own hardware. They are not vendor-supported operation, and a contract that depends on them has no support owner. The correct label is observed-community-only. Table 3 below converts the evidence into a disposition vocabulary for the quote and the acceptance schedule.

DispositionMinimum evidenceAcceptance deliverablesRetest triggers
Supported-and-provenSigned vendor statement naming exact SKU, server, firmware, driver and OS, plus a passing ledgerLedger for every ordered pair, host-staged control, engine log, correctness report, test code and configuration hashesDriver, firmware, kernel, engine, NCCL or BIOS change
Supported-but-unprovenSigned statement without the buyer's test runSame tests run before payment releaseAs above, plus any workload change
Observed-community-onlyA working result on similar hardware, no vendor statementWritten acceptance that no vendor owns the path; fallback plan to host-staged or replicasEvery module or engine update
UnsupportedVendor statement that the pair or class is not supportedDesign for host-staged operation or independent replicasNew vendor statement
UnknownNo statement locatedWritten request; keep out of the performance planOn receipt of a statement

Only the first two rows justify designing a cluster around direct access. The rest are fallbacks with labelled risk, and the alternates should be decided in advance: a different server with a vendor-documented switch topology, a different GPU class, independent replicas, or a measured host-staged configuration accepted on its own latency and throughput numbers.

Matched-run workload template

A capability matters only if it changes a required service level. The workload test should repeat realistic prefill and decode requests at the buyer's concurrency and context lengths, alternate configurations (peer path on, host-staged control) in randomized order, discard a warm-up, and run enough repetitions to report a mean and an interval for each metric. Required fields:

  • Workload definition. Model, precision, context length, concurrency, request mix, quality metric and the latency percentile named in the service-level objective.
  • Configuration identity. Ledger row identifiers, engine flags and all environment variables.
  • Repetitions. Independent runs per configuration and the interval statistic used.
  • Outcome. Qualified requests per service group, not only average tokens per second.

The vLLM forum post is a useful model of disclosure. It states a limitation of one timed aggregate per cell with no CI [9], says differences of a few percent may be run-to-run noise until multi-rep confirmation [9], and verifies that the transport change was real rather than a silent no-op [9]. Demonstrating that the path changed and separately demonstrating that the application changed is the right order of evidence.

Figure 02
Peer-access acceptance test sequence
  1. 01Pair query in both directions

    Record each ordered pair's result with driver and CUDA versions.

  2. 02Read, write and integrity transfers

    Use the allocation types the application uses, in buffer sizes that bracket its message sizes.

  3. 03Bidirectional and concurrent transfers

    Run both directions together, then run every pair the application uses simultaneously to check for contention.

  4. 04Host-staged control

    Repeat with the peer path disabled by the documented means for the layer under test, and archive both runs.

  5. 05Engine transport capture

    Start the real engine with logging that prints transport per channel, and save the lines.

  6. 06End-to-end correctness

    Compare model outputs on a fixed prompt set with a reference configuration.

Data Analysis and Evidence

Link rates and direction conventions

PCIe numbers are published in different conventions, so every comparison must state direction and decimal units. A reprint of the PCI-SIG PCIe 5.0 release gives 32 GT/s and up to 128 GB/s via x16 [52], and an analyst quoted in the same reprint gives 64 GB/second for PCIe 5.0 [52]; these are consistent only if 128 GB/s is the two-direction total. PCI-SIG's PCIe 6.0 release states 64 GT/s and up to 256 GB/s via x16 [53] without a direction word, while its PCIe 7.0 release says 128.0 GT/s and up to 512 GB/s bi-directionally via x16 [54]. The PCIe 8.0 draft announcement states 256.0 GT/s and up to 1.0 TB/s bi-directionally via x16 [55]; as a draft it is not a purchasable generation.

Source discrepancy: NVIDIA's L40S page lists PCIe Gen4 x16 as 64 GB/s bidirectional [25], whereas an RTX A5000 study describes approximately 32 GB/s on a sixteen-lane PCIe Gen 4 link and labels it bidirectional [56]. Both cannot describe the same quantity under the conventions above. Contracts should say "GB/s per direction" or "GB/s aggregate" explicitly, use decimal GB for link rates, and reserve GiB for buffer sizes. NVIDIA's NVLink page describes sixth-generation NVLink as 3 TB/s bidirectional [57].

Published measurements and their limits

The following measurements illustrate magnitudes. Each is tied to named hardware, most are preprints, and none is a 2026 procurement benchmark for another configuration.

  • Host-staged versus P2P bandwidth. On an eight-GPU RTX 5090 platform with a modified driver, a 2026 preprint reports one-way bandwidth improving from 31.5 GB/s to 56 GB/s [8] and bidirectional bandwidth from 32 GB/s to 111 GB/s [8]. It also reports that communication latency decreased drastically from 14.3 microseconds [8] to a sub-microsecond value.
  • Host-staged contention. ThunderEP, a September 2026 preprint, states that neither of its systems supports GPUDirect P2P access between any pair of GPUs [58] and that communication between GPUs must be staged through host memory [58]. A GPU communicating alone reaches 56.5 GB/s on PCIe Gen5 [58], and each GPU's bandwidth falls to 35.5 GB/s [58] when a second GPU under the same host bridge communicates.
  • NVLink versus PCIe at decode sizes. A four-card RTX A5000 study reports that realized NVLink and PCIe bandwidths are close at its message sizes [56], and that the fast bridge slows the collective by a factor of 1.73 at 512 KiB with the direct path off [56]. The authors say that putting all four cards on one shared fast link would require a switch fabric [56], so these are measured points, not a full separation of effects.
  • Host-mediated design. CoMoE states that all cross-GPU traffic must be host-mediated, bounced through host DRAM [13], and reports performance approaching NVLink-capable A800 GPUs at only 23.4% of the hardware cost [13] on a server with two Intel Xeon Platinum 8575C CPUs [13]. This is an author-reported result for a mixture-of-experts workload, not a statement about dense tensor-parallel models.
  • Overhead even on fast links. TokenWeave reports that distributed inference of LLMs can incur overheads of up to 20% [59], so communication cost is non-trivial even where a fast path exists.

Communication volume and a limited screen

Tensor parallelism generates frequent collectives. A characterization study states that each transformer layer generates two Allreduce operations [60], the original Megatron-LM paper describes only two all-reduces in the forward path and two in the backward [61], and a later Megatron paper notes that tensor parallelism is best within a node due to its expensive all-reduce communication [62]. The characterization study concludes that pipeline parallelism suits bandwidth-constrained environments [60]. That is why engine choice, and not only link rate, belongs in the purchase decision.

As an explicitly limited screen, serial communication time is the sum over messages of latency plus bytes divided by measured effective bandwidth:

T_comm = sum over i of ( latency_i + bytes_i / effective_bandwidth_i )

This ignores overlap, contention and the collective algorithm, so it bounds one serial term and is not a model of total inference time. For all-reduce, nccl-tests multiplies algorithm bandwidth by 2(n-1)/n to obtain bus bandwidth, which is 1.0 for two GPUs, 1.5 for four and 1.75 for eight, so numbers from different group sizes are not interchangeable.

(Hypothetical Example) Assume a dense decoder with 80 layers, hidden size 8,192, 16-bit activations and a decode batch of 32 tokens. Each all-reduce moves 32 x 8,192 x 2 = 524,288 bytes, and with two all-reduces per layer a step has 160 messages. Suppose a host-staged configuration measured 15 microseconds latency and 10 GB/s effective bandwidth, and a direct path measured 5 microseconds and 25 GB/s (all four are assumed inputs, not measurements). The staged message costs 15 + 52.4 = 67.4 microseconds and the direct message 5 + 21.0 = 26.0 microseconds, giving 10.8 ms versus 4.2 ms of serial communication per step, a difference of 6.6 ms. Whether that matters depends on the measured step time and latency objective. If both configurations meet the objective, or replicas meet demand without tensor parallelism, the direct path releases no GPU. If only the direct path meets it, whole service groups can be sized from measured qualified requests per group: divide required qualified requests per second by the measured figure per group, round up, and compare both configurations on that integer. A percentage gain does not necessarily release a purchased GPU.

Figure 03
Host-staged bandwidth per GPU in ThunderEPGB/s
Source: ThunderEP

Implications and Future Directions

Procurement language should follow the dispositions. The RFP response should contain the ledger template, the vendor's signed statement per disposition, and the retest triggers. The vendor should state who owns a regression after a driver, BIOS or engine update; without that, a "supported" cell is an opinion.

Retest after every layer change. The peer-access API page and the programming guide carry separate revision dates. Documentation, drivers and engines change on different cadences, and vLLM's custom all-reduce behavior depends on the version in which its checks live [37]. The acceptance record should name versions, not generations.

Interconnect generations are moving. PCI-SIG describes a tradition of doubling IO bandwidth every three years [54]. A faster link narrows the bandwidth gap between host-staged and direct paths but does not remove the extra copy or the contention found on shared host bridges, as the ThunderEP topology of GPUs distributed across three PCIe host bridges shows [58]. Designs that work around absent P2P, such as CoMoE, whose transfer kernel occupies SMs because it moves data with loads and stores [13], suggest software mitigation will keep improving, which strengthens the case for measuring rather than assuming.

Isolation and performance need a joint owner. Because bare-metal and virtualized requirements diverge, one security policy cannot be inherited across both. The platform owner, security owner and model-serving owner should sign the same page.

Unknowns remain unknown. Vendor statements were not located for several SKUs, and no hardware tests were run. A table that fills those cells from a neighbouring generation would be less trustworthy than one that says so.

Frequently Asked Questions (FAQs)

Which consumer GPUs support P2P over PCIe, and what about the RTX 4090?

No current vendor support table was located for this report. One NVIDIA developer forum post relays Engineering feedback that peer to peer is not supported on the RTX 4090 and says drivers should not report such a configuration as peer capable [5]; the post dates from February 2023. Community reports of working peer transfers on consumer cards rely on patched kernel modules and belong in the observed-community-only row. Treat any consumer SKU as unknown or unsupported until the buyer holds a current vendor statement.

Why does cudaDeviceCanAccessPeer return false?

The call queries whether a device may directly access a peer device [2]. Documented reasons for a negative or unusable result include topology and platform settings such as bare-metal IOMMU, which CUDA and the display driver do not support for PCIe peer-to-peer transfers [1]. The query is directional, so check both orderings. The manual page read for this report does not list every cause, so the vendor's release notes are the record of reference.

How is peer access checked with nvidia-smi topo -p2p?

The manual describes the option, as discussed above, as showing P2P status between all GPUs for a given capability. Record the output with the driver version and treat it as one ledger input, because vLLM performs a real access test precisely since a framework-level query can disagree with reality [16].

Can multi-GPU inference run without NVLink over PCIe, and what setup is best for local inference?

Yes, but engine behavior changes. Independent replicas need no GPU-to-GPU path [38], and llama.cpp's default layer split is pipelined [40]. The evidence reviewed supports no single best setup. The sound order is to size the model and context to one GPU first, since distributed inference is probably unnecessary if the model fits on a single GPU [35], use replicas when throughput is the aim, and use tensor or pipeline groups only when the model requires them, with the disposition table deciding which path to rely on.

How much does PCIe bandwidth affect tensor-parallel inference?

It depends on message size, group size and the collective. Published results range from near-parity at small decode sizes in one four-card study [56] to large one-way bandwidth gains with P2P on one eight-GPU server [8]. Neither transfers to another server. TensorRT-LLM's documentation says its PCIe all-reduce optimization improves performance for large message sizes in PCIe-based topologies [42], which marks message size as a deciding variable.

Is a PCIe switch needed?

NVIDIA's reference architecture says a switch may not be needed for low-cost inference servers, while NVIDIA's developer blog calls switches critical for inter-GPU bandwidth [30]. The answer follows the workload: if tensor-parallel groups span many GPUs, ask the vendor for the switch topology and the ACS configuration it ships with.

Conclusion

Multi-GPU PCIe peer access is a chain of conditions, and a purchase should be accepted only when every link has an owner and a record. A product page, an x16 slot, a topology listing or a successful capability query supports one link at most. The vendor must state support for the exact configuration, the runtime must show directional capability, the engine log must show the transport that actually ran, and the application test must show that the change matters for the service level and the number of GPUs bought.

The disposition vocabulary of supported-and-proven, supported-but-unproven, observed-community-only, unsupported and unknown keeps those conditions visible in the contract. Dated documentation, preprints with explicit hardware assumptions and a clearly labelled forum experiment all inform the plan, and none replaces the acceptance ledger. Where evidence is missing, the right entry is unknown, and the right fallback is a measured host-staged configuration or independent replicas, accepted on their own numbers.

External Sources (62)

About

GPUSmith

Plan and build private AI compute with GPU Smith. We connect workload requirements with GPU hardware, networking, deployment and operating decisions so your infrastructure fits the work it must perform.

GPU Smith provides independent engineering and research for private AI infrastructure. We help technical buyers and operators reason about GPU workload sizing, hardware selection, procurement, deployment and inference operations, from the compute system to the networking, power and cooling requirements around it.

Start with the workload

Useful infrastructure decisions begin with the models, data, throughput, latency and operating constraints a team actually has. GPU Smith helps connect those requirements with system design and deployment choices. The goal is a privately controlled AI environment whose capacity and operational demands are understood before a purchasing decision.

Hardware and supplier research

Our public reference library covers GPUs, complete systems and networking components. The server-vendor directory and manufacturing research help teams investigate suppliers, compare options and follow sources behind technical claims. We distinguish manufacturer specifications from measured benchmarks and advertised capabilities from tested configurations. Reference pages are research resources, not a live inventory listing or a binding equipment quote.

Deployment and operations

GPU Smith's engineering scope includes infrastructure integration, networking, power, cooling and the practical operation of inference workloads. Our published articles explain the tradeoffs behind deployment, procurement and ongoing operations so teams can ask better questions and document decisions.

Work with GPU Smith

Explore the hardware library, GPU references, networking references and vendor research. Contact GPU Smith with your workloads and deployment constraints to discuss a project and confirm current scope, pricing and availability.

Inclusion of a manufacturer or product in our research does not imply a partnership, certification or endorsement.

Disclaimer

This document is provided for informational purposes only. No representations or warranties are made regarding the accuracy, completeness, or reliability of its contents. Any use of this information is at your own risk. GPUSmith shall not be liable for any damages arising from the use of this document. This content was generated with assistance from artificial intelligence tools, which may contain errors or inaccuracies. Readers should verify critical information independently. All product names, trademarks, and registered trademarks mentioned are property of their respective owners and are used for identification purposes only. Use of these names does not imply endorsement. This document does not constitute professional or legal advice. For specific guidance related to your needs, please consult qualified professionals.