CUDA Driver Toolkit Container Compatibility Gate

Executive Summary

The production question is not whether a container has “CUDA” in its tag. It is whether five independently versioned layers form a supported and tested path: GPU and architecture, host kernel driver, container runtime injection, image user-space CUDA libraries, and framework or workload binaries. The CUDA Toolkit and the NVIDIA driver are separate products ^[1], while a TensorFlow GPU container can require only the host driver, not a host Toolkit installation ^[2]. Consequently, nvidia-smi is evidence about the active driver and its maximum supported CUDA user-mode interface, not proof of the Toolkit, runtime libraries, or framework build inside an image. NVIDIA defines the CUDA value shown by nvidia-smi as the latest version supported by the driver ^[3].

As of September 20, 2026, the broad minor-version ranges are CUDA 11.x on driver 450 through below 525, CUDA 12.x on 525 through below 580, and CUDA 13.x on 580 or later ^{[4] [5]}. Those are eligibility ranges, not correctness guarantees. Framework constraints can be narrower: JAX’s CUDA 13 route requires at least a Linux 580 driver (Source: docs.jax.dev), and ONNX Runtime requires matching CUDA and cuDNN major versions when sharing PyTorch libraries ^[6]. PyTorch 2.14 itself publishes three distinct CUDA wheel variants, 12.6, 13.0, and 13.2 ^[7].

The recommended release gate is therefore inventory, classify, canary, decide, and retain rollback. Pin the image by digest because a digest identifies invariant content ^[8], save host and in-container inventories, test device visibility, framework initialization, one kernel path, PTX/JIT if used, and a representative inference. Approve only when every applicable matrix and canary passes. Defer an eligible but untested exception. Fail an unsupported range, platform, or library combination. Kubernetes creates a new Deployment revision when a rollout is triggered ^[9], so the prior digest and Pod template provide a concrete rollback target. This is a production upgrade gate, not a promise that nominal compatibility guarantees application correctness.

Introduction and Background

CUDA compatibility is frequently compressed into one comparison: container CUDA version versus the “CUDA Version” line in nvidia-smi. That shortcut discards the layers that determine whether an application actually starts, loads the intended libraries, executes compiled kernels, performs PTX just-in-time compilation, and produces valid results. Backward compatibility, minor-version compatibility, and forward compatibility solve different version relationships. A container adds another boundary, because it normally carries application libraries while depending on the host for the kernel driver and selected user-space driver components.

The practical decision for platform teams is whether a new framework wheel or CUDA image may use an already-qualified fleet driver. A driver update often has a larger operational blast radius than an application image change, so decoupling them has value. The mechanism and its limitations must be identified rather than inferred. A framework’s own checks can reject an otherwise plausible environment, as JAX documents when installed CUDA libraries are not sufficiently new (Source: docs.jax.dev).

This report treats the release as a chain of evidence. GPU Smith is an adjacent engineering advisor, not a CUDA vendor or cloud provider. Its published scope includes capacity planning, telemetry, failure-mode analysis, and operating procedures ^[10]; that posture makes a dated acceptance record more appropriate than a universal version promise. The same site says private AI infrastructure is specified, integrated, and validated against written acceptance criteria ^[11]. Those first-party statements explain the gate-oriented perspective, but they do not replace NVIDIA, framework, or orchestrator documentation.

The resulting method answers all common versions of the query: the minimum driver for a Toolkit family, whether a CUDA container can run on a host driver, when the cuda-compat package applies, why a framework matrix can override a base CUDA pass, and how to upgrade CUDA in production with measurable rollback criteria.

Key Changes

Five version layers replace the one-number model

The first change in operator practice is to inventory five layers, not one. The host and image contribute different parts of the path. TensorFlow’s container instructions make the boundary concrete by requiring the host GPU driver without requiring the host Toolkit ^[2].

Table 1 defines the minimum operator inventory and the question each field answers.

The table shows why two nodes with identical nvidia-smi headers can behave differently: their image digest, injected capabilities, framework build, or workload compilation path may differ. Conversely, a host does not need a matching Toolkit installation merely because the container includes a particular runtime. The official TensorFlow container guidance explicitly separates its host-driver prerequisite from a host Toolkit ^[2].

Backward, minor-version, and forward compatibility are separate branches

Backward compatibility is the least surprising branch: an application built using an older Toolkit runs on a newer driver. This is still subject to the application, library, and architecture requirements. ONNX Runtime, for example, separately requires matching CUDA and cuDNN major versions for shared PyTorch libraries ^[16].

Minor-version compatibility is a newer runtime or application within the same CUDA major family on an older driver that meets the family minimum. It has three important caveats:

• Limited feature set: a feature introduced in a newer Toolkit can also require newer driver support and return cudaErrorCallRequiresNewerDriver ^[17].

• PTX limitation: applications that rely on PTX compilation do not work on the older driver through minor compatibility alone ^[18].

• Architecture targeting: NVIDIA requires a target architecture argument to nvcc for this path ^[19].

Forward compatibility uses a cuda-compat package containing selected user-mode driver libraries so a newer CUDA application can operate with an older base driver. Installing the package alone does not configure the loader to locate those libraries ^[20]. It is also not a general system-driver upgrade: NVIDIA says the forward-compatible path is for CUDA only ^[21]. Treat it as an explicit exception with a package version, supported GPU class, loader configuration, test result, and exit date.

Containers package user space, not driver independence

The NVIDIA Container Toolkit requires the host NVIDIA GPU driver ^[22]. It configures an engine such as Docker or containerd so the NVIDIA runtime can expose devices and driver libraries. Docker provides GPU resources through the --gpus flag ^[23], while the runtime hook exposes the matching device nodes and libraries ^[13].

The image flavor matters. NVIDIA’s official CUDA image documentation says the Container Toolkit is required to run CUDA images with Docker ^[24]; its devel flavor adds headers and development tools on top of the runtime flavor ^[25]. A production inference image should not be assumed to contain build tools merely because a development image does.

Implementation Considerations and Process Changes

Build an attachable inventory before making a decision

Collect inventory on the exact canary node before starting the candidate container. The following commands produce small evidence files without changing the driver:

nvidia-smi --query-gpu=uuid,name,driver_version --format=csv,noheader > host-gpus.csv
nvidia-smi -q -x > host-nvidia-smi.xml
cat /proc/driver/nvidia/version > host-kernel-driver.txt
nvidia-container-cli -V > container-toolkit.txt
docker version --format '{{json .}}' > docker-version.json
docker image inspect --format '{{json .RepoDigests}}' registry.example/image@sha256:DIGEST > image-digests.json
docker image inspect --format '{{json .Config.Labels}}' registry.example/image@sha256:DIGEST > image-labels.json

Formatted image inspection is an official way to view image labels ^[26]. Pulling by digest fixes the exact image version ^[27], which makes the canary and rollback artifacts reproducible.

Then collect evidence from inside the candidate image. A minimal framework-neutral check is:

docker run --rm --gpus all registry.example/image@sha256:DIGEST nvidia-smi --query-gpu=uuid,name,driver_version --format=csv
docker run --rm --gpus all registry.example/image@sha256:DIGEST sh -lc 'ldconfig -p | grep -E "libcuda|libcudart|libnvidia-ptxjitcompiler"'
docker run --rm --gpus all registry.example/image@sha256:DIGEST sh -lc 'test -r /usr/local/cuda/version.json && sed -n "1,80p" /usr/local/cuda/version.json'

Do not interpret the host’s displayed CUDA version as the runtime version. Record host information separately from the framework build and runtime discovery. PyTorch documents torch.version.cuda as a build inspection command ^[28], while TensorFlow says its build-information values are compile-time static ^[15].

For a compact JSON attachment, a Python collector can preserve command output and failures without silently substituting missing values:

import json
import os
import subprocess

image = os.environ["CUDA_GATE_IMAGE"]
commands = {
    "host_gpu_csv": ["nvidia-smi", "--query-gpu=uuid,name,driver_version", "--format=csv,noheader"],
    "kernel_driver": ["cat", "/proc/driver/nvidia/version"],
    "toolkit_cli": ["nvidia-container-cli", "-V"],
    "image_digests": ["docker", "image", "inspect", "--format", "{{json .RepoDigests}}", image],
    "container_gpu_csv": ["docker", "run", "--rm", "--gpus", "all", image,
                          "nvidia-smi", "--query-gpu=uuid,name,driver_version", "--format=csv,noheader"],
}
result = {}
for name, command in commands.items():
    run = subprocess.run(command, text=True, capture_output=True)
    result[name] = {"returncode": run.returncode, "stdout": run.stdout, "stderr": run.stderr}
print(json.dumps(result, indent=2, sort_keys=True)

Run it as CUDA_GATE_IMAGE=registry.example/image@sha256:DIGEST python3 collect.py > cuda-gate.json. A nonzero return code remains evidence rather than being erased. In Kubernetes, also save the Node object, Pod specification, Pod status, and Deployment history. Pod status contains per-container status records ^[29], which ties a result to what the orchestrator actually launched.

Apply a deterministic pass, defer, or fail tree

Use the following tree for every GPU architecture and node pool, not merely one representative machine:

1. Fail inventory completeness if the loaded driver, GPU identity, Toolkit or runtime used at build time, immutable image digest, framework build, and rollback digest are not known.

2. Classify the relationship. If the application Toolkit is older than the host driver’s capability, evaluate backward compatibility. If it is within the same CUDA major family and above the paired driver, evaluate minor-version compatibility. If it crosses a major-family minimum on an older driver, evaluate the exact cuda-compat row and supported platform.

3. Fail unsupported combinations. Fail when the driver is below the documented family minimum without an applicable forward-compatibility package, the GPU class is excluded, or any framework, cuDNN, inference server, or operating-system matrix rejects the combination.

4. Defer conditional combinations. Defer when PTX/JIT is required on a minor-compatibility path, a new driver-coupled feature is needed, loader configuration for cuda-compat is unproven, the image is not digest-pinned, or only an import test has run.

5. Run the canary sequence. Require device visibility, framework initialization, a real allocation and kernel, PTX/JIT when applicable, representative inference with output checks, and telemetry review.

6. Pass only the tested tuple. Approval applies to the recorded GPU architecture, host driver, kernel module, Container Toolkit, digest, image runtime, framework build, compatibility mechanism, and workload test. It does not automatically approve another image tag or node pool.

This conservative tree distinguishes eligibility from approval. An eligible minor-version path is a canary candidate. It is not yet a production pass.

Test progressively, including the path an import misses

The canary should progress from cheap diagnostics to workload evidence:

• Device exposure: verify the intended UUIDs are visible inside the container. PyTorch’s torch.cuda.device_count() returns the available GPU count ^[30].

• Framework initialization: capture torch.cuda.is_available(), TensorFlow physical devices, jax.devices(), or ONNX Runtime providers. JAX documents jax.devices() as the list of devices for a backend (Source: docs.jax.dev), and ONNX Runtime exposes installed execution providers ^[31].

• Allocation and kernel: allocate device memory and execute a small operation. TensorFlow’s documented matrix-multiplication smoke test confirms execution by logging GPU:0 ^[32].

• Architecture coverage: compare the device capability with architectures compiled into the framework. PyTorch exposes its compiled architecture list ^[33].

• PTX/JIT path: force the actual dynamic-kernel or extension path when the workload uses it. Persist exact error names and descriptions alongside the image digest and test input.

• Representative inference: use the real model shape, precision, batching behavior, plugins, and output checks. PyTorch recommends benchmark settings representative of real use cases ^[34], and ONNX Runtime exposes the actual installed execution providers for capture ^[31].

• Orchestrator state: verify readiness and a terminal canary Job result. Kubernetes states that a Pod is ready only when all containers are ready ^[35], and Jobs expose Complete or Failed terminal states ^[36].

Define rollback before the canary

Rollback should be an explicit predicate, not an improvised response. Trigger it for any initialization or unsupported-PTX error, missing expected GPU, framework provider fallback, output mismatch, new error class, or a local telemetry regression outside the team’s predeclared baseline. The baseline itself is deployment-specific and should not be invented from a vendor compatibility table.

Record the prior image digest and Deployment revision. Kubernetes rolls back the Pod template portion of a Deployment ^[37], so separately version any ConfigMap, Secret, model repository, or host change not contained in that template. A driver rollback is a distinct maintenance operation and should not be conflated with image rollback.

Data Analysis and Evidence

The quantitative core is a version-range crosswalk, not a benchmark. Compatibility documentation establishes supported states, while the canary establishes application behavior. No throughput improvement should be inferred from a compatibility pass.

Table 2 summarizes the broad driver-family logic documented on September 20, 2026. Exact Toolkit release notes and downstream support matrices still control each candidate.

The numerical boundary is useful for triage. It is insufficient for approval because a single major family contains multiple Toolkit releases and features. A family floor does not necessarily enable every later driver-coupled feature. Likewise, a CUDA-level pass does not make cuDNN major versions interchangeable: ONNX Runtime binaries built with cuDNN 8.x are not compatible with cuDNN 9.x ^[41].

The evidence bundle should make every decision reproducible. Table 3 is a compatibility-gate worksheet that can be copied into a ticket or comma-separated-value file.

The worksheet turns “driver mismatch” into a bounded diagnosis. Driver initialization, architecture targeting, and PTX compilation are separate failure classes and lead to different remediation paths. Exact error codes and messages should therefore be retained verbatim in the evidence bundle, together with the framework’s device and provider inventory ^[31].

Case Studies and Real-World Examples

These are dated worked examples, not evergreen approvals. Each assumes the cited rules as published on September 20, 2026, and each still requires the full canary.

Driver 535 with a CUDA 12.8 image (Hypothetical Example)

A Linux node on driver 535 and an application built within CUDA 12.x falls inside the documented 525-to-below-580 range ^[4]. The initial state is therefore canary, not automatic pass. If the candidate is JAX, its documented CUDA 12 wheel is built with CUDA 12.3 and accepts local CUDA 12.1 or later (Source: docs.jax.dev), but that framework fact does not remove the driver canary. A PyTorch canary should also record whether CUDA is available to the running process ^[44].

• Approve candidate status when the workload uses supported compiled targets, the framework and library matrices pass, and all canary stages succeed.

• Defer if a custom extension emits PTX at runtime and the older driver must JIT it, because minor compatibility alone does not cover that PTX path.

• Fail if the application depends on a driver-coupled feature introduced after the installed branch and returns the documented newer-driver error.

The rollback target is the previously qualified image digest. Azure’s registry guidance states that a manifest has a unique SHA-256 digest ^[14]; no host change is needed merely to reverse the application image.

Driver 570 with a CUDA 13.0 image (Hypothetical Example)

Driver 570 is below the broad 580 floor for CUDA 13.x minor compatibility (Source: docs.jax.dev). A plain minor-version argument therefore fails. The operator may evaluate the exact CUDA 13.0 forward-compatibility package only if the GPU and platform are supported, the package-to-base-driver matrix marks the pairing compatible, the loader uses the supplied libraries, and the full canary passes.

• Fail by default without a documented and tested forward-compatibility path.

• Defer when the package is installed but library resolution has not been demonstrated. Installation evidence is not loader evidence.

• Approve the exception only for the exact recorded tuple, with an owner and a scheduled path back to the normal driver branch. Record the framework’s actual device list, such as the devices returned for a JAX backend (Source: docs.jax.dev).

Driver 580 with a CUDA 13.4 workload (Hypothetical Example)

Driver 580 meets the broad CUDA 13.x minor-compatibility threshold, but CUDA 13.4 features may require R615 or later ^[39]. The application owner must identify whether those features are used. A JAX deployment provides an additional check because its CUDA 13 route also sets a Linux 580 minimum (Source: docs.jax.dev). TensorFlow offers a separate pre-initialization physical-device inventory for its own canary ^[45].

• Canary an existing code path that stays within the supported minor-compatible feature set.

• Defer when feature use is unknown or a framework release note requires a newer branch.

• Fail the older-driver path when the application explicitly requires a feature coupled to the later branch.

This example demonstrates why the major-family floor and the exact release notes must both appear in the approval record. For ONNX Runtime, the installed execution-provider list is an additional attachable check ^[31].

Implications and Future Directions

Compatibility policy should be maintained as code-like data rather than prose in a runbook. Store version ranges, exception packages, image digests, test definitions, and decision outcomes in a reviewed repository. Regenerate the crosswalk whenever the fleet driver, CUDA family, framework wheel, cuDNN major, base image, or GPU architecture changes.

Cloud and managed-platform behavior must remain explicit. GKE Autopilot manages driver installation together with node provisioning and scaling ^[46], while certain Amazon EKS optimized images include the host driver, CUDA user-mode driver, and Container Toolkit ^[47]. Managed installation changes who supplies a layer, not whether that layer must be inventoried.

Several process changes reduce future upgrade cost:

• Qualify driver branches as fleet products. Record the operating systems, kernels, GPU architectures, partition modes, and node images covered by each branch.

• Publish framework build manifests. Include CUDA variant, cuDNN major, compiled architectures, extensions, and whether PTX or runtime compilation is expected.

• Pin every production image. Tags are discovery aids; promotion and rollback should use digests. Docker explains that digest pulling guarantees invariant image content ^[8].

• Separate eligibility from performance. A compatibility test establishes that a path functions. PyTorch’s guidance calls for thread settings representative of real use cases ^[34], but local thresholds must come from the service objective and qualified baseline.

• Expire exceptions. A forward-compatibility deployment should have an owner, scope, evidence bundle, and planned return to a normally supported base driver.

• Retest narrow matrices first. cuDNN, framework, inference server, operating-system, and architecture support can be narrower than CUDA’s driver rule. ONNX Runtime’s cuDNN-major constraint is one concrete example ^[41].

The long-term goal is not to eliminate driver upgrades. It is to make an application release independent when documentation and tests support that choice, and to make a fleet upgrade deliberate when they do not.

Frequently Asked Questions (FAQs)

What CUDA version does nvidia-smi show?

It shows the driver’s supported CUDA ceiling, not necessarily an installed Toolkit or the runtime inside a container. Inventory the image libraries and framework build separately. PyTorch’s documented torch.version.cuda command inspects the package build ^[28].

What is the minimum driver version for a CUDA Toolkit?

Use the exact Toolkit release notes and Table 2’s dated family-level crosswalk. Feature, platform, and framework requirements may raise the broad floor. JAX, for example, documents a Linux 580 minimum for CUDA 13 (Source: docs.jax.dev).

Does a CUDA container need CUDA installed on the host?

It needs a compatible NVIDIA host driver and a configured container runtime. It generally does not need the matching Toolkit installed on the host. TensorFlow’s container guide states that only the NVIDIA GPU driver is required on the host ^[2].

Can a newer CUDA container run on an older driver?

Sometimes. Within a documented major family, minor-version compatibility can apply above the family minimum, with feature and PTX limits. Across a major-family floor, a supported forward-compatibility package may apply. Official CUDA images still require the NVIDIA Container Toolkit for Docker ^[24], and the exact host, GPU, package, image, and workload must pass the gate.

Does forward compatibility replace a driver upgrade?

No. It is a scoped user-mode compatibility mechanism, not a general upgrade of the host driver stack. Use it as a controlled exception and plan a return to a normally supported driver. The rollback image should remain digest-pinned because Docker states that a digest guarantees invariant image content ^[8].

Why can CUDA pass while a framework fails?

Frameworks and libraries add their own build and support constraints. JAX validates installed CUDA library versions and reports an error when they are too old (Source: docs.jax.dev). ONNX Runtime requires compatible CUDA and cuDNN major versions when sharing PyTorch libraries ^[16].

How should CUDA be upgraded in production?

Inventory all five layers, pin the candidate and rollback images by digest, classify the compatibility mechanism, run a canary on every affected GPU and node-pool class, preserve results, and promote gradually. Updating a Deployment’s container image triggers a rollout ^[48], which provides a revisioned application path, but host-driver changes require a separate maintenance and rollback plan.

What should make the gate defer rather than fail?

Use defer for a potentially supported state with incomplete evidence: unknown PTX/JIT use, unpinned image, unverified loader path, missing framework matrix, or an import-only test. Use fail when authoritative documentation rejects the combination or a mandatory canary returns an error, output mismatch, provider fallback, or readiness failure.

Conclusion

CUDA driver, Toolkit, and container compatibility is a layered release decision. The host driver exposes the kernel interface and supported CUDA ceiling. The container supplies most application user space. The framework wheel and native extensions add narrower CUDA, cuDNN, architecture, and PTX requirements. None of those facts can be replaced by the single CUDA number in nvidia-smi.

A defensible production gate records the GPU architecture, active driver and kernel module, Container Toolkit configuration, immutable image digest, in-image runtime, build Toolkit, framework variant, compatibility mechanism, PTX requirement, canary output, and rollback image. It then distinguishes three outcomes. Pass means every applicable matrix and representative test succeeded for the exact tuple. Defer means a documented path may exist but the evidence is incomplete. Fail means the combination is unsupported or a required test did not succeed.

The broad family floors provide a fast first screen, but exact release notes and downstream matrices remain decisive. Minor-version compatibility reduces unnecessary fleet-wide driver changes, while forward compatibility offers a narrower exception for supported platforms. Both are valuable when treated as documented mechanisms rather than blanket promises. Digest-pinned images, staged tests, preserved error codes, and a rehearsed rollback turn that documentation into an auditable uptime control.