Triton 26.08 Upgrade Runbook: CUDA 13.4, Canary and Rollback Testing

Executive Summary

Triton 26.08, which maps to Triton Inference Server 2.72.0, should be treated as a canary-first release for private AI fleets, not as a routine tag substitution. NVIDIA published the corresponding release on August 31, 2026 ^[1], and the release documentation was last updated that day ^[2]. NVIDIA's 26.08 documentation gives conflicting CUDA versions: its contents list names CUDA 13.3.4.1, while its driver section and container version table identify CUDA 13.4.1. Treat this as an unresolved documentation discrepancy and verify the pulled image before deployment; the container table lists TensorRT 11.2.1.2. CUDA's own notes say 13.x applications have minor-version compatibility on drivers at version 580 or later, while CUDA 13.4 features require an R615-series or later driver ^{[3] [4]}. A production gate should therefore require both a host-driver check and an inspection of the pulled image, rather than trusting any single line of the release page.

The upgrade is most compelling for fleets affected by dynamic-batcher queue starvation, broken preserve_ordering, incorrect model-readiness reporting, non-batching TensorRT CUDA graph execution, or vLLM JSON parsing. Those are documented corrections, not evidence of a throughput gain. Operators should expect no performance improvement by default and should compare the canary with a simultaneously running 26.07 control. Google Site Reliability Engineering guidance explicitly recommends comparing canary and control signals (Source: sre.google), while Prometheus documents why histograms, rather than averaged precomputed quantiles, support aggregate percentile analysis ^[5]. OpenTelemetry likewise defines histograms for statistically meaningful values such as request duration ^[6].

The strongest reason to defer or isolate is backend-specific. NVIDIA states that the Triton TensorRT-LLM backend container is not included in 26.08 ^[7]; the compatibility table stops that image family at 26.07. The 26.08 vLLM image moves from a 0.24.0-based build to a 0.27.1-based NVIDIA build, but explicit model control with tensor parallelism greater than one still requires the Ray executor. Upstream vLLM identifies RayDistributedExecutor as its Ray-based distributed executor ^[8]. Ray itself says isolation must be enforced outside the cluster ^[9]. Multi-GPU vLLM should therefore be isolated behind a trusted network boundary and canaried separately from conventional TensorRT, ONNX Runtime, and OpenVINO models.

The production decision is straightforward. Upgrade conventional backends after every model passes load, readiness, ordering, queue-drain, CUDA graph, output, latency, error-rate, memory, and rollback tests. Pin 26.07 for a TensorRT-LLM fleet until NVIDIA publishes a compatible 26.08 image or the operator validates a reproducible custom build. Isolate vLLM and Ray workloads by backend, model-control mode, and tensor-parallel width. Pin the exact OCI digest, because a digest identifies immutable content ^[10] and Docker can pull that exact version ^[11]. Rehearse the rollback before traffic promotion. The result is a testable decision, not a release-note interpretation.

Introduction and Background

This report is a Triton 26.08 migration guide for platform owners deciding whether the container is safe for a production fleet as of September 19, 2026. It focuses on the bundled stack, backend compatibility, current constraints, acceptance tests, and the upgrade rollback procedure. It does not infer benchmark gains from corrected behavior, and it does not treat a successful container start as production acceptance.

Triton is often only one layer in a serving path. A production change can simultaneously alter the server, TensorRT, CUDA user-space libraries, ONNX Runtime, OpenVINO, vLLM, Python, and the container filesystem. The 26.08 release also raises the supported stack from 26.07's Triton 2.71.0, CUDA 13.3.4.1, and TensorRT 11.1.0.106 ^[12]. That is a compound dependency change, even when model files and client code are unchanged.

The operating posture used here has three principles. It follows the four monitoring signals of latency, traffic, errors, and saturation (Source: sre.google):

• Separate compatibility from correctness. A supported GPU and driver can launch the container, but only workload tests establish correct outputs and stable operation.

• Separate server-wide from backend-specific approval. TensorRT, vLLM, TensorRT-LLM, ONNX Runtime, Python, and OpenVINO do not need to receive the same decision.

• Make rollback measurable. A rollback is not complete until the pinned prior image is serving, models are ready, error and latency signals recover, and the elapsed restore time is recorded.

GPU Smith is an adjacent engineering adviser, not a Triton vendor. Its published approach derives the serving stack from workload models and throughput targets ^[13] and frames acceptance testing against written criteria ^[14]. Its method also issues operating procedures and an as-built documentation package ^[15]. That posture fits this upgrade: the decision belongs to measured workload evidence, not a generic claim that the latest image is better.

Key Changes in Triton 26.08

Exact bill of materials and the CUDA 13.4 discrepancy

The immutable GitHub release identifies 2.72.0 as the server release for NGC 26.08, and GitHub associates it with commit 4bfa701 ^[16]. NVIDIA's container table identifies Ubuntu 24.04, CUDA 13.4.1, and TensorRT 11.2.1.2. Its driver section also says the release is based on CUDA 13.4.1 ^[17], and the separate CUDA deep-learning release repeats that value ^[18].

The 26.08 release page lists CUDA 13.3.4.1 in its contents list but says the release is based on CUDA 13.4.1 in the driver section and lists 13.4.1 in the container version table. This is an unresolved documentation discrepancy; verify the pulled image before deployment rather than labeling either entry stale.

For the standard image, the documented package set includes cuDNN 9.25.0, NCCL 2.30.7, OpenUCX 1.21.0, OpenMPI 5.0.10, DALI 2.2.0, nvImageCodec 0.8.0, ONNX Runtime 1.28.0, OpenVINO 2026.3.0, DCGM 4.6.1-1, and vLLM 0.27.1 ^[19]. ONNX Runtime's own compatibility guidance says CUDA 13.0 builds are compatible across CUDA 13.x minor versions, but it treats the cuDNN major version as a hard boundary ^{[20] [21]}.

Fixes that justify a canary

The 2.72.0 release corrects several behaviors that can materially affect production reliability:

• Dynamic batching. The scheduler could leave requests queued after its waiting-consumer count drifted; 2.72.0 corrects that starvation path ^[22].

• Ordered delivery. preserve_ordering now reserves the completion-queue position when the request is enqueued, restoring arrival-order semantics ^[23].

• Readiness. Unresolved models now report ready=false, and non-availability errors are returned to the caller ^[24].

• TensorRT CUDA graphs. The release fixes CUDA graph execution for non-batching models ^[25].

• vLLM input. JSON input parsing is corrected in the vLLM backend.

• Python clients. The upper version bound on grpcio is removed, reducing client-package pinning friction.

These are reasons to test 26.08 if a fleet encountered the affected paths. They are not a basis for publishing a throughput percentage. A corrected queue can eliminate stalls while leaving steady-state throughput unchanged. The canary must therefore preserve the same hardware, model, precision, request distribution, concurrency, and measurement window as its 26.07 control.

Packaging changes and absent components

The standard 26.08 image remains described as supporting TensorFlow, PyTorch, TensorRT, ONNX, and OpenVINO models ^[26]. NGC describes signed tags as carrying a digital signature that verifies the artifact has not changed since signing ^[27]. vLLM has a separate 26.08-vllm-python-py3 variant; its compatibility row pins Python 3.12.3 and NVIDIA's 0.27.1-based build, CUDA 13.4.1.012, driver 615.61, and a listed size of 10.65 GB ^[28].

TensorRT-LLM differs. There is no published 26.08 TensorRT-LLM backend image, and the compatibility table's latest row for that variant is 26.07. NVIDIA's compose tool also does not currently support composing vLLM or TensorRT-LLM backends ^[29]. The backend repository points operators to a source-build guide ^[30]. That makes a custom image a separately controlled artifact, not a drop-in equivalent to NVIDIA's absent image.

Compatibility and Backend Risk Register

Hardware floor, drivers, and artifact identity

Triton 26.08 supports CUDA compute capability 7.5 or later, covering NVIDIA Turing, Ampere, Hopper, Ada Lovelace, and Blackwell families ^[31]. That is the image-level floor, not proof that every backend, model format, precision, or kernel path behaves identically on every family. OpenVINO's requirements also note that GPU drivers are not bundled with its toolkit ^[32].

The preflight should record:

• GPU identity: model, compute capability, total memory, Multi-Instance GPU configuration, and topology.

• Host stack: driver branch and exact version, kernel, container runtime, and NVIDIA Container Toolkit version.

• Image identity: registry, tag, platform architecture, pulled digest, and signature state.

• Model identity: repository commit or object version, model checksum, configuration checksum, precision, and backend.

• Traffic identity: request corpus checksum, arrival pattern, concurrency, sequence behavior, and warmup policy.

For Linux amd64, NGC exposed the standard 26.08 artifact digest as sha256:71a23d2f27001af58c242380fe05c9d46d5dbc79913d62c60de72543964267df, with an 8.27 GB compressed artifact dated August 31, 2026 ^[33]. Operators must confirm the platform-specific digest at pull time, because the fetched NGC page did not expose one multi-architecture manifest-list digest. Docker prints the digest after a completed pull ^[34], Kubernetes distinguishes immutable digests from movable tags ^[10], and OCI guidance says retrieved content should be verified against its descriptor digest ^[35].

Backend compatibility matrix

Table 1 summarizes the backend decision before any performance test. “Included” means NVIDIA publishes a corresponding image or documents support, not that every model has passed acceptance.

The matrix argues against a fleet-wide binary decision. Conventional TensorRT and ONNX Runtime models can proceed to a canary after host compatibility passes. vLLM needs its own network, process-lifecycle, and multi-GPU tests. TensorRT-LLM needs a packaging decision before runtime testing even begins. Triton's platform matrix expressly warns that not every backend supports every platform ^[39].

vLLM, Ray, and explicit model control

The documented 26.08 constraint is precise: with explicit model control, the default distributed executor does not support vLLM tensor-parallel sizes greater than one. NVIDIA's workaround is to set distributed_executor_backend to ray in model.json. The vLLM backend also requires multi-GPU instance groups to use KIND_MODEL, and the configured GPU ID count must equal tensor parallel size multiplied by pipeline parallel size ^[40].

That workaround changes the operational boundary. Ray recommends a controlled, isolated network for cluster components ^[41], and says token authentication supplements rather than replaces network isolation ^[42]. Ray documents built-in token authentication beginning with version 2.52.0 ^[43]. Therefore:

• TP=1: test the default executor and explicit load/unload path.

• TP>1: test only with the documented Ray setting, correct KIND_MODEL, and a matching world size.

• Network: allow Ray and vLLM peer traffic only inside the trusted serving segment.

• Teardown: verify that the Triton parent and every vLLM or Ray child process exits during unload, shutdown, and rollback.

• Memory: capture allocated, reserved, and peak memory. PyTorch distinguishes tensor-occupied memory from allocator-reserved memory ^[44].

Implementation Considerations and Process Changes

Preflight and staged migration

A safe Triton 26.08 upgrade begins with an inventory, not a deployment. Build a row for every model, backend, control mode, GPU family, precision, and parallel width. Freeze a representative input corpus with expected outputs. Pin the current 26.07 artifact and confirm it can still be pulled from the private registry before introducing 26.08. GPU Smith's published method emphasizes verification at every stage ^[45].

Use these gates in order:

1. Artifact gate: verify the architecture-specific digest, registry signature state, software bill of materials, and internal vulnerability policy.

2. Host gate: confirm compute capability 7.5 or later and a driver that meets the selected CUDA 13.4 feature path.

3. Cold-start gate: launch with no production traffic; record server start time, model load time, and readiness transitions.

4. Functional gate: compare outputs to the frozen corpus, including malformed inputs and backend-specific shapes.

5. Scheduler gate: exercise mixed request sizes, priorities, queue limits, timeouts, and ordered responses.

6. Resource gate: compare GPU memory, host memory, CPU, GPU utilization, queue depth, and process count with the control.

7. Traffic gate: send a small production slice while maintaining a simultaneous 26.07 control.

8. Rollback gate: restore the prior digest and prove readiness, output correctness, and service-level recovery inside the approved restore-time objective.

Triton supports NONE, EXPLICIT, and POLL model-control modes, with NONE as the default; NVIDIA does not recommend POLL for production ^{[46] [47]}. Test the mode actually used in production. A successful model load or unload API request returns HTTP 200 ^[48], but that response is only one checkpoint. The model must subsequently become ready and answer an inference correctly.

Canary validation matrix

Table 2 turns the Triton 26.08 upgrade validation tests into observable go/no-go criteria. Each relative threshold must be filled from a service objective or the matched 26.07 control. No universal latency or error percentage is assumed.

The table should be executed on each production hardware and model class. Kubernetes readiness removes an unready pod from Service load balancers ^[52], while a startup probe can suppress readiness and liveness checks until startup succeeds ^[53]. This distinction prevents a slow model load from being mistaken for a dead server.

Rollback procedure and evidence

Rollback should be rehearsed before the 26.08 canary receives traffic. Keep both image digests and both model-repository states available. Docker supports pulling an exact version by digest ^[11], and OCI image configuration is immutable because changing it changes the computed image identifier ^[54].

The rollback sequence is:

1. Stop promotion: freeze additional traffic and configuration changes.

2. Drain the canary: stop new routing, allow accepted requests to finish within the drain objective, then capture final metrics and logs.

3. Restore image: deploy the recorded 26.07 digest, not merely the movable 26.07-py3 tag.

4. Restore models: return the model repository and configurations to their recorded versions.

5. Verify readiness: require startup completion, server readiness, model readiness, and a known-answer inference before routing traffic.

6. Verify recovery: compare errors, p95 latency, queue delay, GPU memory, host memory, and process count with the pre-change control.

7. Record restore time: measure from stop-promotion to the first sustained healthy window.

Kubernetes retains Deployment rollout history by default, subject to its revision-history limit ^[55], and can target a particular revision during rollback ^[56]. Helm likewise provides history and rollback commands (Source: helm.sh) (Source: helm.sh). Argo Rollouts can retain a rollback window for a fast return to a retained ReplicaSet ^[57]. None of these mechanisms substitutes for a recorded container digest, model version, and post-restore inference test.

Data Analysis and Evidence

Quantified component change from 26.07

Table 3 isolates version and artifact changes that can be measured without implying performance. Version numbers are compatibility inputs, not benchmark results.

The vLLM size comparison comes from NVIDIA's compatibility rows: 26.07 lists 10.28 GB ^[59] and 26.08 lists 10.65 GB. The calculation is (10.65 - 10.28) / 10.28, rounded to one decimal place. It says nothing about throughput, latency, or memory use at inference time.

Measurement plan and rollback worksheet

Triton exposes a pending-request gauge and cumulative queue-duration counters. AIPerf's Triton integration describes nv_inference_pending_request_count as backend queue depth ^[60] and captures a metrics baseline before warmup ^[61]. Perf Analyzer can run a fixed schedule through its input JSON once, which is useful for matched timing ^[62]. Model Analyzer reports p95 latency and maximum GPU memory from a run.

Record one worksheet row per hardware, model, precision, concurrency, and backend configuration:

• Identity: server digest, model checksum, GPU model, GPU count, precision, batch policy, concurrency, and corpus checksum.

• Demand: offered requests per second, accepted requests, completed requests, and test duration.

• Quality: expected-output matches, mismatches, timeouts, and application error rate.

• Latency: median, p95, p99, first-token latency for streaming models, and end-to-end latency.

• Queue: maximum pending requests, cumulative queue-delay delta, drain time, and rejected requests.

• Resources: peak GPU memory, reserved memory, host memory, CPU, GPU utilization, and child-process count. PyTorch can capture a complete allocator-state snapshot ^[63].

• Lifecycle: load time, unload success, readiness transition time, shutdown completion, and restore time.

• Decision: threshold source, pass or fail, reviewer, timestamp, and evidence location.

OpenTelemetry defines histograms as appropriate for statistically meaningful values such as request duration ^[64] and describes them as compressed representations of a measurement population ^[65]. Prometheus identifies the 0.95 quantile as the 95th percentile ^[66] and cautions that averaging precomputed quantiles is statistically unsound ^[67]. The canary window must be at least as long as the aggregation interval; Google SRE guidance says metric intervals should be equal to or shorter than canary duration (Source: sre.google).

Thresholds should come from the service-level objective and control distribution. Argo Rollouts deliberately leaves success and failure values to the operator ^[68]. A useful gate says, for example, “p95 may not exceed the approved control-relative margin for three consecutive matched windows,” with the margin filled by the service owner. Publishing a universal 5% or 10% cutoff without workload evidence would create false precision.

Decision Framework: Upgrade, Pin, or Isolate

Upgrade now

Approve 26.08 for a backend slice only when all of the following are true:

• Platform compatibility passes: supported GPU floor, correct driver path, and verified platform digest.

• Every deployed model loads: no reliance on an untested auto-complete path or stale serialized-engine assumption.

• Correctness passes: the frozen corpus matches within the model's declared numerical tolerance.

• Scheduler tests pass: ordered responses remain ordered and the queue drains after overload.

• Resource tests pass: peak and steady memory, process count, and utilization remain inside workload-specific limits.

• Rollback passes: the prior digest and model state return to service within the restore-time objective.

This path is strongest for conventional TensorRT, ONNX Runtime, and OpenVINO deployments that benefit from the documented reliability fixes and have complete canary coverage. The decision record should retain the same canary and control signals (Source: sre.google) and the written acceptance criteria ^[69].

Pin 26.07

Keep 26.07 when the fleet depends on NVIDIA's prebuilt TensorRT-LLM image, when the host driver cannot meet the chosen CUDA 13.4 path, when a serialized engine does not pass load and correctness tests, or when the rollback rehearsal misses its objective. Pinning is also appropriate when the affected fixes provide no operational value and the qualification cost is not justified yet.

Pin both image and models. A tag alone is insufficient because tags can move, while the digest identifies content ^[10]. Docker's pull documentation confirms that digest selection specifies the exact version ^[11]. Preserve the test artifacts so that a later 26.08 retry starts from the failed gate rather than repeating the entire qualification.

Isolate by backend

Use a separate deployment, node pool, or serving segment for vLLM and Ray. The separate unit should have its own digest, network policy, readiness probes, process-lifecycle checks, dashboards, and rollback. For TP>1, require the documented Ray executor and world-size mapping ^[8]. For TP=1, do not add Ray merely for consistency unless the deployment design already requires it. Ray's own guidance places security and isolation outside the Ray cluster ^[9].

Argo Rollouts can define a rollback window that fast-tracks return to a retained ReplicaSet ^[57]. gRPC also defines a standard health service, and clients with health checking enabled wait until it reports healthy ^{[70] [71]}. These controls help make isolation executable, but Triton model readiness still needs an application-level known-answer request.

Implications and Future Directions

Triton 26.08 demonstrates why monthly inference containers need backend-level release governance. The server tag is only the visible coordinate for a larger graph of CUDA, TensorRT, libraries, Python packages, model formats, process managers, and monitoring behavior. A future release process should automatically diff that graph, store the pulled digest and software bill of materials, and generate test obligations from changed components.

Three process changes are particularly durable:

• Artifact evidence should be first-class. Store the architecture-specific digest, signature result, and internal registry copy beside the deployment manifest. OCI guidance calls for digest verification when content comes through an untrusted source ^[35].

• Model lifecycle should be continuously tested. Load, unload, readiness, and teardown are operational behaviors, not one-time installation checks.

• Backend decisions should be independent. A missing TensorRT-LLM image should not block an unrelated ONNX Runtime canary, and a passing TensorRT model should not authorize multi-GPU vLLM.

The CUDA line discrepancy also supports a machine-verified preflight. Release notes are human-readable summaries. The deployment record should capture what the actual image reports and compare it with the support matrix. When they differ, the release remains in canary until the operator identifies which value controls compatibility.

Future NVIDIA updates may publish a 26.08 TensorRT-LLM image or revise current constraints. Until that happens in fetched primary documentation, the production record should say “not published” rather than infer a version. Likewise, the current tensor-parallel workaround and Ray boundary should remain explicit acceptance items until the release notes remove them.

Frequently Asked Questions (FAQs)

Is Triton 26.08 compatible with CUDA 13.4?

Yes, the controlling NVIDIA sources identify CUDA 13.4.1. One contents-list line says 13.3.4.1, so validate the pulled image and host driver. CUDA 13.x minor compatibility begins at driver 580 for existing applications, while new CUDA 13.4 features require R615 or later. ONNX Runtime separately documents CUDA 13.x minor compatibility for its CUDA 13.0 builds ^[20]. Use the feature path, not the most permissive theoretical floor, to choose the production driver.

Does Triton 26.08 contain a TensorRT-LLM backend image?

No. NVIDIA explicitly says that image is not included, and the compatibility table stops at 26.07. An operator can remain on the 26.07 variant or build a custom 26.08 artifact from documented source instructions, but the latter requires its own reproducibility, compatibility, and rollback qualification.

What are the most important Triton 26.08 breaking changes?

The release is better described as a compound compatibility change than as one universal breaking change. CUDA, TensorRT, ONNX Runtime, OpenVINO, DCGM, and vLLM versions move; the TensorRT-LLM image is absent; and multi-GPU vLLM under explicit control retains a Ray-specific workaround. Treat each as a migration gate.

What belongs on the CUDA 13.4 upgrade checklist?

Record the GPU compute capability, driver version, architecture-specific image digest, CUDA version observed inside the image, TensorRT version, backend package versions, model and configuration checksums, and test-corpus checksum. Then run load, readiness, output, ordering, queue, graph, memory, multi-GPU, teardown, and rollback tests.

How should rollback success be measured?

Measure from the promotion stop to a sustained healthy window on the prior digest. Require the old model repository to be restored, all required models to be ready, a known-answer inference to pass, and error rate, p95 latency, queue delay, memory, and process count to return within their approved objectives. Helm defines rollback as returning a release to a previous revision (Source: helm.sh), while Argo requires explicit success and failure values for analysis ^[68]. A standard gRPC health service can hold requests until healthy status is reported ^[71]. Record the restore time and evidence location.

Conclusion

Triton 26.08 is suitable for a controlled, backend-specific canary, not an unqualified fleet-wide rollout. Its 2.72.0 server fixes directly address queue starvation, ordered response delivery, readiness reporting, selected CUDA graph execution, and vLLM input parsing. Those corrections are operationally valuable, but they do not establish a throughput gain.

The upgrade gate has three hard branches. Conventional TensorRT, ONNX Runtime, OpenVINO, and Python models may advance after host compatibility, output, scheduler, resource, and rollback tests pass. Multi-GPU vLLM should be isolated, configured with the documented Ray workaround under explicit control, and tested for network boundaries, world-size correctness, memory, and clean teardown. TensorRT-LLM fleets should pin 26.07 unless they deliberately qualify a reproducible custom 26.08 build.

The final production record should name the exact digest, GPU, driver, model, precision, concurrency, control mode, thresholds, results, and restore time. That evidence converts a release note into a defensible go, pin, or isolate decision. The practical rule is conservative: approve only the backend and configuration that produced the evidence. Do not let one passing model, one GPU generation, or one control mode authorize a different serving path. If any required row is blank, the appropriate state is still canary or defer.