How to Qualify GPUDirect Storage with GDSIO: Runbook

Executive Summary

To qualify NVIDIA GPUDirect Storage (GDS), an operator must establish three separate facts: that the installed software and the exact file path are eligible, that functional graphics processing unit (GPU) reads and writes succeed, and that the measured run uses the intended data path. GDS is a direct memory access (DMA) path between storage and GPU memory that can avoid a CPU bounce buffer. ^{[1] [2]} An installed libcufile or a successful gdsio process is insufficient evidence by itself, because a compatible POSIX path can still complete I/O through system memory. ^{[3] [2]} A qualification report should identify the path as P2PDMA, NVFS, or compatibility, using the modes exposed by the installed GDS release, then attach counters or diagnostic logs for the run. NVIDIA added explicit p2pdma, nvfs, and compat reporting to gdscheck in GDS 1.16. ^{[4] [5]}

The practical sequence is: inventory GPU, kernel, CUDA, storage client and mount; map the GPU to the nonvolatile memory express (NVMe) controller or remote direct memory access (RDMA) network interface card (NIC); run gdscheck; perform read and write checks on a disposable test file; compare matched gdsio workloads; and preserve the logs. For local block storage, NVIDIA currently documents EXT4 and XFS as the supported filesystems. ^{[6] [7]} For a distributed filesystem, the filesystem vendor's version and mount requirements are an additional gate: BeeGFS calls for RDMA across its nodes, WEKA can revert to standard I/O when requirements are unmet, and VAST calls for an RDMA-enabled mount. ^{[8] [5] [9]} The matrix below makes those checks path-specific rather than treating a cluster as globally enabled.

Benchmark numbers are evidence about the tested workload, not portable promises. Keep block size, file set, read/write mix, GPU, placement, threads and test duration fixed between paths. gdsio uses O_DIRECT file descriptors by default; Linux alignment rules still vary by filesystem and kernel, and a network filesystem may cache data at the server even when client page cache is bypassed. ^{[10] [11] [7] [12]} Record throughput together with process CPU utilization, errors, POSIX/bounce counters and the mode declaration. NVIDIA documents POSIX pool counters as system-memory bounce-buffer statistics for compatibility mode. ^{[13] [5]} A path passes production qualification only when the observed mode and integrity results meet the operator's written acceptance criteria at the required workload read rate; this report does not assign a universal throughput threshold.

Introduction and Background

A GPU pipeline can be stalled by data delivery even when the GPU itself has adequate arithmetic capacity. The useful question is narrower than whether a node has GDS installed: can this particular GPU read this particular mounted path through the intended data path, at the needed rate, without an unobserved fallback? This report is a runbook for storage, platform and machine-learning infrastructure engineers evaluating local NVMe and distributed filesystems. It treats eligibility, functional correctness and performance as separate gates. GDS moves data between storage and GPU memory through DMA and is designed to avoid the extra CPU-memory staging of a conventional path.

The site publishing this guide, GPU Smith, describes its private AI infrastructure engagements as governed by written acceptance criteria and an as-built documentation set. ^[14] That advisor perspective belongs in the acceptance method: a storage path should be signed off with a reproducible record, not with a vendor slide or an isolated peak number. Neither a GPU model name nor a filesystem brand establishes the mode used by an individual file. NVIDIA's current tool reporting distinguishes p2pdma, nvfs, and compat; that distinction must be recorded for the installed release and target path.

The runbook dates version-sensitive claims to September 23, 2026. NVIDIA's docs span several GDS and CUDA generations, and the installed package's help output is the final authority for executable name and switches. Filesystem claims in this article are scoped to the vendor documentation version named in each row. The controlled test should use a scratch mount and a capacity limit agreed with storage operators, because directory-based gdsio writes create files before a read test can consume them. ^[15] The method applies to a single test node or a larger cluster rollout. Subsequent sections explain what to collect, what to run, and how to interpret a result without turning a single benchmark into a procurement forecast.

GDS Paths and the Qualification Decision

Decide whether the workload needs a direct path

Start with the pipeline's required sustained read rate, not an interface's advertised peak. In a (Hypothetical Example), a training job that consumes eight input streams at 0.5 GB/s each requires 4 GB/s of delivered reads by simple multiplication; the stream rate is a planning input, not a measured GDS result. Add the job's burst pattern, data transforms, checkpoint writes and acceptable CPU headroom. If the ordinary path already exceeds demand and CPU use is acceptable, GDS may still be useful for scaling, but the immediate production decision can be based on the measured margin. If CPU staging is the suspected constraint, compare both throughput and CPU time under the same data set. pidstat -u provides per-task CPU utilization, whereas fio's grouped CPU values are averages across jobs. ^{[16] [17]}

The data path has three operational categories. Direct GDS uses a supported PCI peer-to-peer DMA path or NVIDIA's filesystem integration. Compatibility mode permits cuFile calls to complete through POSIX I/O and CPU memory staging. Ordinary application I/O uses the application's normal POSIX path and its explicit transfer to GPU memory. The latter is a useful comparison, but it is a different software path. A change in throughput between these categories cannot alone prove which layer caused it. NVIDIA notes that its GDS statistics do not separately distinguish PCI P2PDMA from nvidia-fs; pair them with gdscheck, topology and device records. ^[18]

Local NVMe is a PCIe device, while networked NVMe can use RDMA or TCP transports; the name NVMe alone does not specify the GPU path. The NVM Express specification lists PCIe, RDMA and TCP transports separately. ^[19] CUDA 12.8 introduced an NVMe P2PDMA configuration that can operate without nvidia-fs.ko when its other requirements are met. ^{[20] [19]} Consequently, an unloaded nvidia-fs module is not a universal failure signal on current local NVMe nodes. For a distributed filesystem, the client's GDS integration, mount protocol, RDMA configuration and storage-side version must all be checked on the actual path. BeeGFS, WEKA and VAST each publish distinct requirements. ^{[8] [21] [9]}

Classification before timing

Before any performance run, write down the intended mode and the evidence that could disprove it:

• Direct path candidate: supported GPU and software, supported filesystem and mount, suitable topology, direct-I/O capable target file, and gdscheck showing the relevant capability. ^{[22] [23] [24]}
• Compatibility candidate: cuFile calls work, but the I/O is served through a POSIX path or a vendor-documented standard-I/O fallback. ^[5]
• Unqualified path: any prerequisite is unknown, functional verification fails, the result changes without a known cause, or the expected mode cannot be corroborated.

This classification deliberately keeps “functional” apart from “direct.” NVIDIA documents managed CUDA memory as an indirect, unregistered-buffer case rather than a normal registered GDS buffer; a successful application read using managed memory therefore does not establish the same path as a registered device buffer. ^[25] Capture the memory allocation type used by the production application, and use the same type in a representative functional test.

Support Matrix and Topology Gate

Version, filesystem and mount prerequisites

Table 1 is a September 23, 2026 evidence matrix, not a universal compatibility guarantee. The installation's CUDA, driver, GDS and filesystem-client versions still need comparison with current vendor release notes. NVIDIA documents EXT4 and XFS for local block device filesystems and gives platform-specific minimum kernels for NVMe P2PDMA in its release notes. ^{[26] [7]}

The matrix distinguishes a vendor's feature statement from an operator's result. In particular, a distributed filesystem's RDMA fabric may be healthy while the GPU data path falls back. WEKA explicitly describes such a reversion, and BeeGFS supplies beegfs-net to verify its storage and metadata connections. ^{[5] [27]} Keep separate rows for every mount, because client configuration and storage target can differ even within one cluster.

Topology and Evidence Gate

Draw GPU to PCIe switch to root complex to NVMe controller or RDMA NIC before selecting a test GPU. NVIDIA lists root complex, switches and physical device placement as factors in delivered GDS performance. ^{[33] [34]} The upstream lspci -t view shows buses, bridges and their connections; nvidia-smi topo -m is called out by BeeGFS for GPU-to-RDMA-NIC topology. ^{[34] [35]} Capture PCI bus addresses rather than relying only on friendly GPU ordinals, which can change across systems. A PCI device's sysfs numa_node file reports its associated nonuniform memory access (NUMA) node or -1 if unknown. ^[36] numactl --hardware shows the system NUMA layout, and remote memory access can carry a performance penalty. ^{[37] [38]}

For each candidate, record the GPU UUID and PCI address, storage device or mount source, NIC address if remote, shared switch/root complex and NUMA node. findmnt --target resolves the filesystem containing the actual test path, avoiding an assumption based on a parent directory. ^[24] Do not change BIOS PCIe settings or unload modules merely to improve a score during qualification; record the as-found configuration, make changes through normal change control, and repeat the identical matrix afterward. The goal is to measure the deployable state.

Use this support and topology checklist before approving a benchmark slot:

• GPU identity: save UUID, model, driver and PCI address for each tested GPU.
• Mode capability: save the complete gdscheck output for the installed release.
• Kernel and CUDA: record exact versions, architecture and the GDS package revision.
• Mount identity: resolve the test file with findmnt --target; record source, type and options. ^[24]
• Block device route: save controller PCI address and the lspci -t tree. ^[34]
• Remote route: save GPU-to-NIC topology and NUMA placement. ^[35]
• NUMA record: retain PCI numa_node values, including -1 when the node is unknown. ^[36]
• RDMA state: for BeeGFS, verify storage and metadata connections, not just link status. ^[27]
• Mount protocol: for VAST, confirm that the tested NFS mount is RDMA-enabled. ^[9]
• Feature exclusions: for WEKA, record whether filesystem encryption is enabled. ^[30]
• File alignment: inspect direct-I/O alignment where statx supports it. ^[39]
• Scratch controls: record available space, file ownership and a cleanup plan.

Functional Verification and GDSIO Protocol

Preflight and an integrity matrix

On each node, collect package versions, kernel, GPU identity, mounted source and options, gdscheck output and the installed gdsio help screen. NVIDIA recommends gdscheck as an installation verification step; current releases can expose supported modes and topology information. Interpret its report alongside the specific file being tested. For local NVMe, note that NVIDIA's current documentation allows P2PDMA without nvidia-fs.ko from CUDA 12.8 in qualifying configurations, while CUDA Toolkit 13.4 changes how nvidia-fs is packaged on non-DGX platforms. ^[40]

Use a scratch directory with known free capacity and permissions. gdsio -D writes one file per worker before directory reads, so a read-only invocation against an empty directory is not a valid integrity test. Choose a block size that meets the mount's direct-I/O alignment; Linux says O_DIRECT restrictions vary by filesystem and kernel, and misalignment may fail with EINVAL or be buffered. ^{[7] [41]} On Linux 6.1 and newer, statx can report file-specific direct-I/O alignment where the filesystem implements it. ^[42] Preserve the exact alignment result if available.

Table 2 defines a functional matrix to run on every candidate mount and selected GPU, using the installed tool's documented switches. Its purpose is to isolate failure, not to rank throughput.

The paired read/write tests matter because a throughput result without verified data can mask a path or workload mistake. NVIDIA says gdsio -V read verification should use files written with the verification option. If an application uses managed memory, report that explicitly and avoid equating its result with a registered device-buffer test. Leave the test files intact until checksums, logs and counters are archived, then remove only the scratch data created for qualification.

Controlled GDSIO Baselines

The installed gdsio help output should be saved in the evidence bundle. NVIDIA's guide defines -I 0/1/2/3 as sequential read/write and random read/write, and documents -w, -s, -i and -T for workers, size, I/O size and duration. ^[43] Record the selected GPU, transfer mode, file or directory target, block size, workers, file size and duration for every run. Do not claim a queue depth that the tool did not expose or measure. If using fio for a supplementary test, inspect achieved I/O-depth distribution rather than trusting the requested depth; fio's own documentation warns that the two can differ. ^[44]

The following is a (Hypothetical Example) command sequence, with an existing scratch directory and GPU index 0. Confirm every switch against the installed --help before execution, choose a file size within the scratch allocation, and save the outputs. NVIDIA's documented -x modes distinguish storage-to-GPU, storage-to-CPU, and storage-to-CPU-to-GPU transfers; -d chooses the GPU index, while -D creates a file per worker on the write pass.

gdscheck.py -p
gdsio -D /mnt/qual-scratch -d 0 -x 0 -I 1 -w 4 -s 8G -i 1M -T 120 -V
gdscheck.py -f /mnt/qual-scratch/testfile
gdsio -D /mnt/qual-scratch -d 0 -x 0 -I 0 -w 4 -s 8G -i 1M -T 120 -V
gdsio -D /mnt/qual-scratch -d 0 -x 2 -I 0 -w 4 -s 8G -i 1M -T 120

The file check needs an existing target file. The -D write creates one file per worker, so replace testfile above with one of the generated filenames shown in the write output. The final -x 2 comparison represents CPU staging and GPU transfer; it is not a forced cuFile compatibility-mode test. Keep a separate compatibility run only when the installed configuration offers a documented, reversible way to force that mode. gdsio exposes worker threads, but the published help does not specify a dedicated queue-depth switch; record worker count and any observed concurrency instead. ^[44]

A minimal sequence is to create data with a verified write, run a verified read on the same geometry, then rerun matched direct, compatible and CPU comparisons. Use three independent repeats as a local protocol choice, not as an NVIDIA standard, and preserve each result rather than only the best. Stabilize concurrent cluster load, storage-side cache state and placement as far as operations permit. fio's ramp_time illustrates the distinction between a warm-up interval and a measured interval; if used, record both. ^[47] gdsio normally opens files with O_DIRECT, which helps avoid a client page-cache benchmark, but O_DIRECT on NFS does not bypass the server's cache. ^[12]

Avoid a global cache drop on shared production nodes. Linux documents that drop_caches discards clean caches and can impose performance costs; it is a testing/debugging control, not a routine performance setting. ^{[48] [49]} Use a fresh file set, a clearly labeled warm-cache repeat or a maintenance window instead. A cold versus warm comparison is useful only if the state is known and repeatable. Capture storage-side cache policy when available and avoid calling a client-cache bypass “end-to-end cold.”

Fallback Detection, Results and Troubleshooting

Prove the mode with independent signals

Treat the requested gdsio transfer type as an input, not a verdict. The direct path should have the expected gdscheck capability, a supported target file and mount, functional integrity, and counters consistent with direct I/O. NVIDIA's gds_stats documentation identifies POSIX pool buffer statistics as system-memory bounce-buffer evidence for compatibility mode. For a short diagnostic run, NVIDIA describes a TRACE-level cufile.log message, cufile IO mode: POSIX. TRACE logging itself can perturb performance, so the diagnostic run should be separate from the timing run. A filesystem vendor's own fallback statement is another clue: WEKA says a requirements mismatch can return its path to standard I/O. ^[5]

Table 3 is a blank local results record. One row is required for each mount, GPU, transfer mode, block size and placement; do not pool different paths into a single average.

This table deliberately separates requested mode, reported mode and measured outcome. A high throughput number does not override a POSIX fallback counter or a verification error. PCIe link rates are capacity context, not target GDS rates: Intel's PCIe table lists 32 GT/s for PCIe 5.0 and a theoretical x16 example near 63 GB/s, while the actual storage route includes other devices and software. ^{[50] [51]}

Failure signatures and rollback

• gdscheck lacks the intended mode: record CUDA, GDS, GPU and kernel versions; compare them to current release notes. NVIDIA's P2PDMA support is conditional, and the nvidia-fs requirement differs by release and platform.
• EINVAL or verification mismatch: inspect file, memory-buffer and offset alignment before changing performance parameters. Linux warns that direct-I/O alignment behavior varies across filesystems and kernels; BeeGFS documents 4 KB alignment for its GDS I/O. ^{[41] [28]}
• CuFile call succeeds but POSIX counters move: label the result compatible; inspect mount support, O_DIRECT, client RDMA state and cufile.log.
• Remote path is unexpectedly slow: compare GPU-to-NIC topology, NUMA placement, client RDMA and mount options. VAST requires an RDMA-enabled mount for its path, and BeeGFS provides a way to check storage and metadata RDMA connections. ^{[9] [27]}
• Read appears implausibly fast: check the file set, duration and client/server cache state. O_DIRECT on NFS bypasses only the client page cache.
• Results vary between repeats: record background jobs, storage occupancy and achieved concurrency before changing software. A requested fio depth may differ from its achieved depth. ^[44]

The rollback is operational: restore the known working application path and configuration, preserve the failed test's logs and version snapshot, and leave direct-path sign-off on hold. Do not convert a forced compatibility result into a production “GDS pass.” If a vendor-specific setting is changed, rerun the full functional matrix for that mount and compare the exact same geometry. This makes the decision auditable even when the final disposition is that ordinary I/O is adequate.

Data Analysis and Evidence

Required rate and useful comparisons

The most useful number is the ratio measured sustained read rate divided by workload-required read rate for the actual placement. In a (Hypothetical Example) with eight simultaneous readers at 0.5 GB/s each, required rate is 8 × 0.5 = 4 GB/s. If the locally measured GDS run delivers 5 GB/s under the matched file set, the observed ratio is 5 / 4 = 1.25. Those inputs are illustrative assumptions, not published vendor benchmarks or a universal acceptance threshold. A production decision should replace them with a trace of the actual pipeline, include variance across repeats, and state the minimum headroom the operator requires for concurrent activity. The arithmetic makes a capacity decision explicit while keeping it separate from the question of whether the direct path is active.

For every mode, report median and range of repeat throughput, transferred bytes, elapsed time, process CPU utilization, verification result and errors. pidstat -u can supply process CPU values; fio's own CPU utilization is averaged across jobs in a reporting group, so the aggregation must be identified. ^{[16] [17]} Keep reads and writes separate: an application that reads training data and writes checkpoints has different service demands. Keep sequential and random workloads separate as well, because gdsio provides distinct selectors for them. If the distributed filesystem uses RDMA, capture its client and network evidence alongside throughput. BeeGFS asks for RDMA to both storage and metadata services; VAST's GDS documentation requires an RDMA-enabled mount. ^{[27] [9]}

Several published numbers are specifications or vendor examples, not local targets. PCI-SIG identifies PCIe 5.0 as 32 GT/s, and Intel's x16 example reaches about 63 GB/s at the link level. ^{[52] [51]} That number is not an expected gdsio output: the storage device, path, PCIe topology, data size and software can all be limiting. WEKA's documentation says RDMA typically becomes advantageous for reads of 32 KB or larger and writes of 256 KB or larger in its architecture; those are vendor-scoped guidance, not a universal GDS threshold. ^[53] BeeGFS specifies 4 KB alignment for its GDS requests, which is an eligibility condition rather than a performance forecast. ^[28] NVM Express released its 2.4 specifications on August 4, 2026, showing why transport and version records should be dated, but a specification revision itself does not qualify a mount. ^[54]

A sound analysis also tests whether the comparator is fair. fio documents direct=1 as usually selecting O_DIRECT, and its CUDA engine distinguishes a cufile path from a POSIX path that copies via a RAM buffer. ^{[11] [55] [2]} This can be useful as a supplementary cross-check, but do not blend fio and gdsio results as if they were the same workload generator. State each tool's version, engine, direct-I/O mode and achieved depth. The outcome is a local performance envelope with an integrity and mode verdict, not a universal vendor ranking.

Implications and Future Directions

Handover evidence

The evidence bundle for each result row should contain:

• Run identity: node, operator, date, start and end times.
• Software manifest: kernel, CUDA, driver, GDS and filesystem-client versions.
• GPU map: UUIDs and nvidia-smi topo -m output. ^[35]
• PCI map: lspci -t, storage controller and NIC addresses.
• Mount proof: findmnt --target output for every test file. ^[24]
• Tool syntax: installed gdscheck and gdsio help output.
• I/O geometry: file list, size, alignment, block size, workers and duration. ^[7]
• Cache state: record client and known server cache conditions separately.
• CPU trace: collect process-level utilization during each timed run.
• Mode proof: retain counters, compatible-path diagnostics and errors. ^[5]
• Integrity proof: retain verification output and any checksum record.
• Disposition: write pass, hold or fallback with the specific acceptance rule.

Requalification after change

The version gate will keep changing. NVIDIA's release notes added explicit mode reporting in GDS 1.16 and describe newer POSIX behavior when required cuFile libraries are absent. ^{[4] [56] [5]} The current local NVMe path can use P2PDMA without nvidia-fs.ko under the documented CUDA 12.8 conditions, while other configurations still depend on the driver and filesystem integration. A static checklist therefore needs an attached version snapshot and a dated support-matrix review, not an undated “GDS enabled” box.

For platform owners, the durable unit of acceptance is GPU plus file path plus mount plus software release. A second GPU behind a different switch, a changed NIC or a new filesystem-client build merits its own row. The Linux PCI tree and NUMA metadata provide a reproducible map of those differences. ^[36] For procurement, the evidence bundle should travel with the proposed storage design: required workload rate, topology, support statements, integrity results, direct and compatible comparisons, CPU use and fallback counters. A vendor's feature announcement can start a conversation, but a dated client/mount support statement and local test complete it. DDN and Dell both describe GDS-related integrations publicly, yet their cited pages alone do not specify this article's tested path. ^{[31] [32]}

GPU Smith states that its engineering recommendation is independent of reseller quota or a cloud service of its own. ^[57] In that advisor role, the practical recommendation is to set acceptance criteria before timing, use a reproducible scratch workload and report an honest hold when mode evidence is missing. This is especially important for distributed filesystems that can serve data correctly after a mode change. WEKA explicitly documents reversion to standard I/O when RDMA or GDS prerequisites are unmet. ^[5] Future revisions of a site runbook should update the versioned filesystem rows, rerun the same test geometry and archive both the prior and new mode evidence; otherwise a performance trend can be mistaken for a hardware change.

Frequently Asked Questions (FAQs)

How do engineers verify that GPUDirect Storage is enabled?

Run the installed gdscheck tool, capture its supported modes and topology, then test the actual target file with verified reads and writes. NVIDIA recommends gdscheck for installation verification and documents mode reporting. Add cuFile statistics or a short TRACE diagnostic to detect a POSIX path; the documented cufile IO mode: POSIX message is explicit. A successful benchmark alone establishes only that I/O completed.

What is a defensible GDSIO baseline command protocol?

Save gdsio --help from the installed package; choose a scratch directory, GPU and supported aligned block size; write verified files; read the same geometry; then run matched direct, compatible and CPU comparators. NVIDIA's guide maps -I 1 to write and -I 0 to read, and warns that -D directory tests need a write first. Record workers, size, I/O size, duration, direct-I/O state, repeats and cache state. ^[11] Treat queue depth as observed or tool-specific rather than inventing a gdsio flag. ^[44]

Does an unloaded nvidia-fs module mean GDS is disabled?

No blanket conclusion follows. NVIDIA documents an NVMe P2PDMA mode from CUDA 12.8 that can omit the module when its requirements are met. ^[20] Check the actual release, GPU, kernel, target file and gdscheck modes. Other paths can still require NVIDIA's filesystem integration; the package and driver state belong in the evidence record.

How should compatibility mode be interpreted?

It is a functional fallback path for cuFile I/O that stages through CPU system memory, so label its results separately from direct GDS. Monitor POSIX pool or bounce-buffer counters and, when necessary, a short TRACE run; neither a high score nor a successful cuFile call overrides those signals. Compare CPU load and integrity under matched conditions before deciding whether the ordinary path is sufficient.

Conclusion

GPUDirect Storage qualification is a sequence of eligibility, integrity, mode proof and workload fit. Begin with the required read or write rate. Inventory the exact GPU, software, filesystem and mount, then map the path through the PCIe and RDMA topology. Run gdscheck, perform verified reads and writes on a disposable file set, and compare controlled gdsio modes with the same geometry. Store throughput beside CPU utilization, errors, cache state and fallback evidence. The support matrix and blank result table make the decision reviewable per path, rather than per product name.

A direct-path pass means that the configured path is supported, the functional data is correct, the measured run has corroborating direct-mode evidence, and the local rate satisfies written workload criteria. A compatibility result can still be an acceptable production choice if the workload meets its own criteria, but it should be named accurately. When evidence is incomplete, the appropriate outcome is a documented hold with the ordinary path preserved and a narrow retest plan. That record remains useful as CUDA, GDS, kernel and filesystem versions change. The archive gives the next software upgrade a fixed comparison point: the same file path, placement and workload can be retested while the former mode, integrity outcome and CPU cost remain visible. A change in a peak score alone is not sufficient sign-off.