GPUDirect RDMA Validation: Topology and Memory Paths

Executive Summary

GPUDirect RDMA validation should establish that a specific GPU and RDMA adapter can register and transfer the intended GPU allocation through a documented direct-access mechanism. NVIDIA's GPUDirect guide identifies a shared upstream PCIe root complex as a key requirement for its documented PCIe peer path. Eligibility therefore begins with device placement, not a network bandwidth test. Linux's peer-to-peer documentation independently explains the restrictions around forwarding between PCIe hierarchy domains. ^{[1] [2]}

A loaded nvidia-peermem module is not the definition of a pass. GPU Operator 26.3 documents DMA-BUF and legacy peer-memory alternatives. Its DMA-BUF prerequisites include an open GPU kernel module, CUDA 11.7 or later, Linux 5.12 or later, and supported Turing-or-newer GPU categories. These are that deployment stack's prerequisites, not a universal compatibility guarantee. The driver README separately warns that the older peer provider and nvidia-peermem cannot operate concurrently. ^{[3] [4]} Distribution packaging also matters: Ubuntu's kernel-team guidance describes its own transition associated with kernel 6.8. ^[5]

The recommended evidence sequence is inventory, topology, resources and security policy, registration, controlled transfer, correctness coverage, then application correlation. Inspect negotiated links rather than capability alone. Reconcile IOMMU and ACS requirements with OEM and security owners: Linux explicitly warns that disabling ACS redirect removes device isolation. ^{[6] [7]} Use host-memory RDMA as a control, then select the intended GPU-memory mechanism explicitly. Current perftest documentation supports content validation, but the maintainer notes that checking may skip completed chunks when it cannot keep up. Retain coverage as well as the final status. ^[8]

There is no measured throughput or application speedup claim in this report. Its PCIe calculations provide context: an x16 link at 32 GT/s with 128b/130b encoding has an encoding-adjusted ceiling of 63.02 GB/s per direction, before protocol overhead. This is not a GPUDirect acceptance threshold. ^{[9] [10]} Finally, a CUDA-aware application can still stage traffic through host memory. A host qualification pass must be followed by evidence of the application's selected path and representative workload measurements. ^[11]

Introduction and Background

Healthy network links do not establish a direct path between a graphics processing unit (GPU) and a network interface controller (NIC). GPUDirect remote direct memory access (RDMA) provides GPU-to-peer Peripheral Component Interconnect Express (PCIe) access, with shared-root-complex constraints in NVIDIA's documented path. Device placement is therefore part of host qualification. ^[1]

This report addresses cluster and network engineers deciding whether a particular GPU, host channel adapter (HCA), and software stack are eligible for that path. It distinguishes a host-side connectivity decision from acceptance of RDMA over Converged Ethernet (RoCE) or InfiniBand fabric behavior. It also separates qualification from process placement and application optimization. Linux's PCI peer-to-peer documentation explains why this matters: traffic can remain below a switch, while forwarding between separate hierarchy domains is generally blocked unless hardware is known to support it. ^[2]

GPU Smith's independent advisory perspective here is to make the acceptance decision reproducible: retain the selected devices, exact software, observed registration mechanism, test output, and limits of the conclusion. The proposed workflow and acceptance categories are engineering recommendations. They are not a vendor certification or a report of tests performed on a supplied cluster.

The evidence reviewed for this report includes vendor documentation, Linux and RDMA implementation documentation, distribution guidance, and maintained test utilities. Commands below are templates requiring local substitution, not executed measurements. No host inventory, logs, or benchmark results were supplied. A completed report should therefore distinguish documented eligibility from locally demonstrated behavior, especially because a CUDA-aware library can accept GPU buffers while staging traffic through host memory. ^[12]

What a GPUDirect RDMA Pass Establishes

Direct transfer describes where payload data moves. It does not imply that the CPU disappears from setup, registration, synchronization, or completion handling. A useful validation record identifies the allocation, export or peer-memory mechanism, RDMA registration, selected devices, and completed operation. The verbs API makes registration a distinct step: ordinary memory registration and DMA-BUF registration have different interfaces, and registration can fail independently of allocation. ^[13]

Use separate acceptance statements rather than an undifferentiated “RDMA passed” label:

• Eligibility: The documented hardware, topology, driver, and operating-system requirements match the intended deployment.
• Device access: The test identity can open the intended GPU and RDMA device.
• Registration: The selected GPU allocation successfully becomes an RDMA memory region.
• Transport: The selected endpoints complete the intended operation using that region.
• Correctness: A supported checker verifies received contents, with its coverage recorded.
• Performance: Separately collected measurements meet a locally approved baseline for the same configuration.

These categories are a proposed reporting structure. Their separation follows the implementation boundary between GPU allocation and export, verbs registration, and transfer validation. Perftest's CUDA implementation explicitly distinguishes allocating GPU memory from requesting a DMA-BUF handle. Its resource code separately reports memory-region registration errors. ^{[14] [15]}

DMA-BUF, Linux's shared direct-memory-access buffer framework, and nvidia-peermem are alternative integration paths in NVIDIA's GPU Operator documentation. Absence of the latter is therefore not an automatic failure. The older nv_peer_mem project documents comparable driver-integrated functionality beginning with R470, which helps explain why historical runbooks use different module names. ^{[3] [16]}

Conversely, a completed application collective is insufficient direct-path evidence. Unified Communication X (UCX) explicitly supports either direct GPU operations or copies through host memory. An application can therefore be functional without demonstrating the mechanism being qualified. The recommended acceptance wording is narrow: “the recorded GPU-memory registration and transfer configuration passed the recorded checks,” followed by the actual evidence and exclusions. ^[11]

Qualify the GPU-NIC PCIe Topology

Begin with stable identifiers. Record each GPU's universally unique identifier (UUID) and complete PCI domain, bus, device, and function address, abbreviated BDF. NVIDIA recommends UUID or PCI bus ID for consistent identification because index ordering can change across reboots. The lspci -D option preserves PCI domain numbers; -t shows the bus tree and -vv exposes detailed device information. ^{[17] [18]}

Use a read-only inventory first. Discover supported query fields from the installed binary rather than copying a query from an unrelated driver release:

nvidia-smi --help-query-gpu
nvidia-smi --query-gpu=index,uuid,pci.bus_id,name --format=csv
nvidia-smi topo -m
lspci -D -t
lspci -D -vv
ibv_devices
ibv_devinfo -d mlx5_0

The NVIDIA manual documents selective properties and CSV output. Red Hat's core RDMA procedure documents adapter enumeration and selected-device inspection. Here, mlx5_0 is a replaceable local device name, not a required name or a statement about the intended adapter. ^{[17] [19]}

Trace the GPU and NIC through intervening switches and bridges to their upstream root ports. A shared non-uniform memory access (NUMA) node is useful locality evidence, but this runbook does not accept that label as a substitute for drawing the PCIe path. HPE's architecture guidance illustrates internal PCIe switches connecting GPUs, NICs, and processors; an actual server's slot map must establish its own arrangement. ^[20]

Table 1 is the proposed eligibility crosswalk. Complete it for each tested pair, retaining the underlying output rather than only a colored status.

The table prevents a locality label from standing in for eligibility. It also makes changes reviewable: a slot move changes the topology record, while a driver change changes the software record. Neither should inherit the other's evidence automatically.

Finally, distinguish link capability from negotiated operation. Altera's PCIe guide identifies LnkCap as advertised capability and LnkSta as negotiated speed and width. Inspect relevant upstream links as well as endpoints; a capability listing alone does not establish the operating path's available bandwidth. ^[6]

Review BAR Resources, IOMMU and ACS

A base address register (BAR) describes a PCI resource window. NVIDIA's management documentation describes BAR1 as mapping GPU device memory for access by CPUs or other PCIe devices and exposes total, used, and free values. Record these values around registration attempts. Do not interpret a GPU's installed memory capacity as proof that the same amount is available through the relevant aperture. ^[17]

NVIDIA's documented PCIe peer path requires disabled or identity/pass-through input-output memory management unit (IOMMU) translation because peers must share physical addresses. Reconcile that compatibility constraint with supported platform security policy before considering a change. ^[1]

Access Control Services (ACS) affect PCIe routing and isolation. Linux explicitly warns that disabling ACS redirect removes isolation between devices and can enlarge IOMMU groups. Its VFIO documentation also explains that topology and interconnect properties influence isolation granularity. An apparent connectivity improvement does not, by itself, establish an acceptable isolation design. ^{[7] [21]}

Before changing policy, use this review sequence:

• Map the affected path: Identify the exact endpoints, bridges, and switch ports.
• Identify the deployment model: Record bare-metal ownership, virtualization, and tenant boundaries.
• Read model-specific guidance: Use the server's supported configuration and firmware documentation.
• Approve the exception: Have the platform and security owners accept any isolation change.
• Limit the change: Record only the devices and settings actually affected.
• Prepare rollback: Preserve the original state and a recovery method before applying it.

The distinction between general constraints and model-specific guidance is material. Dell's PowerEdge XE9680L manual explicitly calls for X2 APIC and operating-system ACS changes for GPUDirect RDMA performance. Separate Dell guidance links ACS state to Virtualization Technology on its discussed platforms. These are scoped OEM instructions, not portable commands for arbitrary servers. ^{[22] [23]}

Retain PCIe Advanced Error Reporting (AER) evidence before and after tests. Linux documents captured-error messages and exposed counters. Use the relevant devices' counter changes and timestamps as supporting evidence; absence of a logged error does not replace successful registration, content checking, or topology qualification. ^[24]

Select and Verify the Peer-Memory Mechanism

Freeze the software inventory before modifying packages: kernel, GPU driver and module flavor, CUDA libraries, NIC driver and firmware, RDMA userspace, test binary, and deployment image. Record whether network drivers come from the distribution or an OpenFabrics Enterprise Distribution (OFED) package. The purpose is to identify one coherent configuration that can be reproduced and rolled back.

Table 2 summarizes documented integration paths reviewed on October 18, 2026. The numerical prerequisites are scoped to the named documentation, rather than presented as universal requirements.

These rows describe different administrative scopes. A distribution's packaging transition, an Operator deployment matrix, and a driver README answer different questions. Their version numbers should remain attached to their sources. A locally installed package may also expose fewer features than a current upstream manual.

For a legacy configuration, NVIDIA documents modprobe nvidia-peermem and warns about the older provider running concurrently. Its README instructs reinstalling the GPU driver when installation before MOFED prevented compilation against the required RDMA APIs. Apply the instructions for the selected release during an approved maintenance change, then retain module-load output and kernel diagnostics. ^[4]

Check access under the actual job identity:

• RDMA enumeration: Inspect the chosen adapter and port using the distribution's supported utilities. ^[19]
• Device permissions: Verify access to the intended /dev/infiniband/uverbsN character device and its assigned group or mode. ^[25]
• Locked memory: Inspect effective ulimit -l; Red Hat documents login-time application of its pinning limits. ^[26]
• Container devices: Expose only required devices through the runtime's supported device controls. ^[27]
• Container limits: Set and verify the intended memlock limit; Docker documents its separate soft/hard limit controls. ^[28]
• Scheduler allocation: Confirm both GPU and RDMA resources; Kubernetes documents distinct device-plugin implementations. ^[29]

Docker documents IPC_LOCK as an addable capability for memory-lock operations. That does not establish that every RDMA container needs it. Check the runtime's effective limits and required operations before adding capabilities. A privileged launch should not become the acceptance procedure merely because it masks an access problem. ^[27]

Run Controlled GPU-Memory Transfer Tests

Choose a peer with a known working host-memory RDMA configuration and use a dedicated test window. Record the selected GPU, adapter, port, global identifier (GID), operation, payload size, queue configuration, and binary version on each endpoint. Keep the same fabric path while changing memory type where possible. This makes the comparison interpretable without claiming that it isolates every possible host variable.

Check the installed utility first. Perftest documents automatic CUDA-header discovery starting with release 25.07; older build instructions may differ. Its maintained manual documents --use_cuda, PCI-bus-ID selection, and --use_cuda_dmabuf. Inspect the actual binary's help rather than assuming a distribution build includes every upstream feature. ^{[8] [30]}

The following command template uses locally chosen test parameters, not an acceptance benchmark. Substitute endpoint-local adapter, port, GID index, and GPU ordinal. Both sides must agree on test semantics; their local device names need not be identical. Perftest documents server/client pairing and these option meanings. ^[31]

# Server: host-memory control
ib_write_bw -d mlx5_0 -i 1 -x 3 -s 65536 -n 10000 --report_gbits

# Client: replace SERVER_IP with the server address
ib_write_bw -d mlx5_0 -i 1 -x 3 -s 65536 -n 10000 --report_gbits SERVER_IP

Repeat the pair with --use_cuda=0 for the selected GPU allocation. For a supported DMA-BUF configuration, also append --use_cuda_dmabuf before the client's server address. For legacy peer-memory testing, omit that DMA-BUF flag and verify the intended module path. These commands select the mechanism explicitly; they do not demonstrate that the local stack supports it. ^[8]

#Server GPU-memory DMA-BUF transfer, if supported
ib_write_bw -d mlx5_0 -i 1 -x 3 -s 65536 -n 10000 --report_gbits --use_cuda=0 --use_cuda_dmabuf

#Client, using its own locally selected devices
ib_write_bw -d mlx5_0 -i 1 -x 3 -s 65536 -n 10000 --report_gbits --use_cuda=0 --use_cuda_dmabuf SERVER_IP

Run the sequence as distinct observations:

• Host control: Establish whether ordinary host-memory RDMA completes.
• GPU allocation: Capture the selected GPU and successful allocation.
• Memory registration: Retain export and registration messages, including failure codes.
• GPU transfer: Complete the same operation using the selected GPU allocation.
• Correctness: Run a supported content checker separately.
• Direction: Exchange endpoint roles and qualify the reverse configuration.
• Repeatability: Repeat the frozen configuration before declaring a local baseline.

Oracle's tutorial shows CUDA-enabled test output containing GPU selection, PCIe information, and allocation. Those lines are useful allocation evidence. Registration still needs its own result: the verbs API separately returns a memory-region pointer or failure, and perftest reports registration errors with diagnostics. ^{[32] [13] [15]}

For builds supporting --data_validation, the current manual requires reliable-connection bandwidth WRITE/READ tests and matching transmit depths of at least 32; -a is incompatible. Run correctness and throughput separately because checking adds overhead. Record mismatches, skipped work, and coverage: the maintainer documents chunks completing faster than the checker can verify. A successful status without coverage context is incomplete evidence. ^{[33] [8]}

Completion is not CUDA visibility. Follow the allocation's documented synchronization protocol before dependent GPU work; NVIDIA warns that concurrent kernels can observe inconsistent DMA writes. ^[1]

Isolate Faults and Preserve the Evidence

Use the earliest failed boundary to direct the next diagnostic step. The following is a proposed fault tree, not a claim that one symptom uniquely identifies one cause:

GPU/RDMA device unavailable
  -> identity, allocation, permissions, scheduler resources
GPU allocation succeeds; registration fails
  -> export capability, peer provider, driver/API compatibility, BAR resources
Host-memory transfer fails
  -> RDMA device, endpoint parameters, connection setup, fabric
Host-memory passes; GPU-memory fails
  -> prioritize GPU registration and the selected host peer path
Transfer completes; content checking fails or lacks coverage
  -> correctness, synchronization, checker configuration, resource contention
Controlled transport passes; application is unsatisfactory
  -> library path selection, collectives, affinity, workload behavior

Registration failure is a particularly useful boundary. Perftest's resource implementation distinguishes a memory-region error and an explicit unsupported DMA-BUF diagnostic. Preserve the exact error rather than collapsing every registration failure into a generic network fault. Successful CUDA allocation has already answered a different question. ^[15]

For application correlation, inspect Message Passing Interface (MPI) and UCX build/runtime capability first. Open MPI documents ucx_info -v, ompi_info, and its runtime CUDA-support query. UCX additionally documents ucx_info -d for available transports and devices. Capability discovery establishes availability, not use by every message. ^{[34] [11]}

Then capture the actual operation. UCX documents UCX_LOG_LEVEL=info and UCX_PROTO_INFO=y for transport/protocol information. For the NVIDIA Collective Communications Library (NCCL), retain debug logs and their subsystem context; NVIDIA cautions that formatting varies with version. Correlate those records with the previously qualified GPU and NIC identities. ^{[11] [35]}

A temporary application-level negative control can strengthen interpretation. NCCL documents NCCL_NET_GDR_LEVEL=LOC as disabling GPU Direct RDMA by distance policy, but its C2C setting can override that policy. On releases supporting it, account for NCCL_NET_GDR_C2C=0 and confirm behavior in logs. Using NCCL_IB_DISABLE is a different question: setting it to 1 disables the IB/RoCE transport itself. ^[36]

Package a qualification record containing:

• Identity: Host, GPU UUID/BDF, NIC BDF, adapter, port, and test identity.
• Configuration: Firmware, kernel, modules, userspace libraries, container image, and test build.
• Topology: Annotated bridges, negotiated links, and effective routing/translation policy.
• Stage results: Allocation, export, registration, transfer, and correctness coverage.
• Measurements: Units, direction, payload, concurrency, repetitions, and local baseline.
• Rollback: Original settings, approved changes, restoration commands, and retest scope.

Data Analysis and Evidence

No measured bandwidth is reported here. The quantitative analysis is an encoding-adjusted PCIe ceiling, intended to check units and impossible expectations. PCI-SIG documents the successive per-lane signaling rates; Intel's PIPE specification documents 128b/130b encoding at 8, 16, and 32 GT/s. These are interface properties, not GPUDirect benchmark results. ^{[9] [10]}

For a one-direction link, calculate:

GB/s ceiling = signaling rate in GT/s × lanes × 128/130 ÷ 8
Gbit/s ceiling = signaling rate in GT/s × lanes × 128/130

The calculation converts signaling into encoded data capacity. It excludes PCIe packet overhead and other system limits. Intel's processor documentation independently associates PCIe Gen4 with 16 GT/s and Gen5 with 32 GT/s, using the same encoding. ^[37]

Table 3 applies that formula to an x16 link. Values are rounded calculations using decimal GB/s, not a throughput target.

The table is derived from the documented signaling and encoding inputs. It supplies no acceptance percentage: a real path includes devices, links, transaction overhead, and the chosen operation. Apply the formula to negotiated width and speed, not merely advertised capability. ^{[9] [10] [6]}

Keep raw signaling, encoding-adjusted capacity, and measured payload throughput as separate fields. A bidirectional sum is also different from a one-direction result. This distinction avoids treating a large advertised aggregate as the amount one transfer should achieve. The local evidence sheet should state direction and reporting units before comparing runs.

Perftest's --report_gbits switches reporting to Gbit/sec rather than its documented MiB/sec default. Preserve the selected reporting mode in the command record. A unit conversion can reconcile two reports without explaining their underlying performance; it should never be used to hide a changed test configuration. ^[33]

For repeated local measurements, retain raw output and report the sample count, median, range, warm-up policy, and whether content checking was enabled. These are proposed reporting choices, not vendor-defined pass thresholds. Define any acceptance band in advance for the frozen configuration. Investigate changes against that baseline before changing congestion controls or collective parameters.

Finally, report content coverage separately from bandwidth. Perftest's maintainer explicitly notes skipped or stale validation work under contention. The appropriate evidence includes how much was checked and any mismatches, rather than only a throughput number or final process exit status. ^[8]

Implications and Future Directions

The useful qualification unit is a versioned GPU-NIC-host configuration, not a GPU model alone. A change to slot placement, firmware, kernel, GPU module, RDMA driver, userspace, container exposure, or application library creates a reason to review the affected boundary. The proposed process is to rerun the smallest relevant checks, then refresh the complete record before wider deployment.

Distribution policy illustrates why current documentation matters. Ubuntu's kernel-team migration notice describes a transition associated with its kernel packaging, and documents signed open-driver variants. That is evidence about Ubuntu's stated deployment path, not a reason to infer identical behavior across distributions or every GPU generation. ^[5]

For containers and schedulers, retain the same evidence boundary:

• Host qualification: Preserve a controlled result outside the workload container.
• Container qualification: Repeat under the intended image, identity, device mappings, and limits.
• Scheduler qualification: Retain allocated GPU and RDMA resource identities.
• Application qualification: Verify the selected transport on representative operations.
• Change control: Remove temporary diagnostic overrides and preserve the final production configuration.

Kubernetes documents vendor drivers and device plugins for GPU scheduling, while its device-plugin framework separately covers high-performance NICs and InfiniBand adapters. The engineering inference is that successful GPU scheduling alone does not demonstrate RDMA-device allocation or a usable peer-memory path. Verify both resources in the launched workload. ^{[38] [29]}

The same caution applies to CUDA-aware communication. Open MPI documents transport- and version-dependent CUDA support, and a documented host-staging fallback exists. After a host qualification pass, application logs and workload measurements remain necessary to determine whether that application uses the path effectively. No connectivity result in this report establishes an application speedup. ^{[34] [12]}

Conclusion

GPUDirect RDMA validation should produce a bounded, reproducible claim. The record should identify the GPU and RDMA device, show their PCIe path, document acceptable translation and routing policy, establish the selected memory-integration mechanism, and preserve successful registration and transfer evidence. Correctness checking and local performance measurement deserve separate results.

The proposed sequence is deliberately layered. First inventory stable identifiers and software. Then inspect the topology and resources. Reconcile IOMMU and ACS requirements with platform support and isolation policy. Verify the selected DMA-BUF or peer-memory path under the intended user and container context. Run the host-memory control, GPU-memory transfer, supported content check, and reverse configuration before interpreting an application collective.

A pass should remain attached to the tested configuration and operation. It should not be expanded into a universal compatibility claim, a fabric certification, or an assumed application improvement. A failure should likewise remain attached to the earliest failed boundary until evidence narrows its cause.

The practical deliverable is the evidence bundle: inventory, topology, configuration, commands, diagnostics, correctness coverage, measurements, and rollback information. With that record, an engineer can explain precisely what worked, what remains unverified, and which next check will reduce uncertainty. That is the point at which fabric tuning and application optimization become better-informed engineering work.