vLLM 0.29 Upgrade Guide: Runner V2 and API Checks

Executive Summary

vLLM 0.29.0 is not a routine package refresh. Released on September 9, 2026, at commit 98dff2a, it makes Model Runner V2 (MRV2) the default for all models ^{[1] [2] [3]}. The correct promotion question is therefore not simply whether 0.29 starts. It is whether each production tuple, meaning model, hardware backend, quantization, attention path, parallelism, speculative mode, Low-Rank Adaptation (LoRA), multimodal behavior, and Key-Value (KV) transfer, runs through the expected runner and still meets an operator-defined contract.

The release artifact choice must also be explicit. It publishes separate CUDA, ROCm, XPU, and CPU distributions, and the fetched CUDA 12.9 image metadata binds its tag to the release commit ^[4]. Pin the resolved image digest or wheel hash, record the host driver and runtime, and preserve the complete dependency lock. Docker documents digest pulls as the way to select an exact image version, while pip hash-checking requires a hash for every requirement ^{[5] [6]}.

Acceptance should combine correctness, capacity, performance, and exposure. Replay the application's real request distribution, compare request schemas and application-visible outputs, then measure time to first token (TTFT), inter-token latency (ITL), end-to-end latency, throughput, error rate, GPU memory high-water mark, and maximum safe concurrency. Aggregate latency with distributions that remain comparable across replicas, a use case Prometheus documents for histograms ^[7]. Do not promote based on pull-request microbenchmarks or exact token identity alone.

Finally, --api-key is not a perimeter. The versioned security guide documents routes that remain credential-free even when the key is configured ^[8]. Production acceptance must enumerate the live route table, test authentication route by route, block direct access, and fail the canary if any unauthorized route is reachable. This aligns with NIST's position that network location alone grants no implicit trust ^[9].

Introduction and Background

This report is an operational vLLM 0.29 migration guide for private-LLM platform engineers, site reliability engineers (SREs), security engineers, and infrastructure owners. It supplies a vLLM Model Runner V2 validation plan, explains the vLLM 0.29 Model Runner V2 default, identifies vLLM 0.29 API changes, defines vLLM API exposure checks, and turns vLLM upgrade compatibility testing into measurable acceptance evidence. Those changes affect correctness, memory use, latency, capacity, and reachable attack surface, so a package-level smoke test is insufficient.

MRV2 is designed around persistent batch state, asynchronous scheduling, GPU-visible metadata, and explicit CUDA graph management. These are meaningful architectural changes, but they are not a promise of a universal speedup. The design page cautions that MRV2 is not yet feature-complete or rigorously tested ^[10].

The practical unit of change control is the complete serving tuple, not the vLLM version alone. Kernel choice, attention backend, model revision, tokenizer or processor, quantization, graph capture, parallelism, speculative decoding, and request mix can all change what executes or what users observe. PyTorch also warns that deterministic algorithms alone do not guarantee full application reproducibility ^[11]. The upgrade plan therefore treats deterministic seeds as a diagnostic aid, not proof that two builds are semantically identical.

GPU Smith is an adjacent independent engineering advisor, not a vLLM vendor. Its published method says that vLLM and Triton are configured to measured requirements and that integration is accepted against written criteria ^{[12] [13]}. That framing fits this decision: promote only when a recorded configuration passes a workload-specific acceptance contract.

Key Changes in vLLM 0.29.0

Release identity and artifact families

The first control is to identify exactly what is under test. The release publishes multiple wheel and container families rather than one interchangeable binary ^[14]:

• CUDA 13.0: default PyPI install and vllm/vllm-openai:v0.29.0 container.

• CUDA 12.9: vllm/vllm-openai:v0.29.0-cu129 container and separate release assets.

• ROCm: versioned ROCm 7.2.3 wheel index and vllm/vllm-openai-rocm:v0.29.0 container.

• XPU: a release-specific XPU wheel and container family.

• CPU: a release-specific CPU container and prebuilt wheel assets.

These labels select a family, not the operator's immutable artifact. Record the registry digest, architecture-specific manifest digest, and wheel SHA256 actually pulled. OCI guidance says retrieved content should be checked against its descriptor digest when consumed from an untrusted source ^[15]. Provenance can supplement that record by describing where, when, and how an artifact was produced (Source: slsa.dev).

Model Runner V2 becomes the default

The central behavior change is simple to state and easy to misapply: MRV2 is the default for all models, but it is not necessarily the effective runner for every workload. The release identifies sequence parallelism, dual-batch overlap, elastic expert parallelism, custom logits processors, and certain speculative-decoding methods among temporary gaps ^[16]. It states that configured unsupported features still fall back to MRV1 ^[17]. The pinned configuration also falls back when Triton is unavailable ^[18].

MRV2 can change resource behavior even when the API surface appears unchanged:

• Persistent state: batch state remains resident rather than being reconstructed as one monolithic per-step input.

• Asynchronous preparation: CPU scheduling and input work can overlap current GPU execution.

• GPU metadata access: include host-resident metadata traffic in profiler and latency comparisons.

• Sampling memory: test logprob-heavy requests as their own memory cohort.

• Graph control: full CUDA graphs are captured and launched through standard PyTorch APIs ^[19].

These mechanisms explain what to measure. They do not justify transplanting a maintainer microbenchmark into a production capacity forecast.

API behavior that needs explicit compatibility checks

The main compatibility risk is rarely a removed URL. It is a small semantic change in tokenization, chat formatting, structured output, tool-call parsing, stop handling, multimodal representation, or errors.

• Chat templates: compare the configured template and rendered prompt; Hugging Face states that a template should match the format used during model training ^[20].

• Multimodal templates: the template can live on the processor rather than the tokenizer, so both revisions belong in the bill of materials ^[21].

• Structured output: the json constraint targets a JSON Schema, so validate the complete schema contract ^[22].

• Schema assertions: object properties are optional unless listed as required, so a weak test schema can pass incomplete output ^[23].

• Logprobs and errors: compare sampled-token probability, alternatives, null fields, and status codes; the compatible contract always returns the sampled token's log probability when requested ^[24].

• Stops and seeds: the compatible completion contract excludes the matched stop sequence from returned text, while seeded sampling remains best-effort rather than guaranteed deterministic behavior ^{[25] [26]}.

Implementation Considerations and Process Changes

Freeze the production tuple before testing

An upgrade test is interpretable only when the old and new tuples are reproducible. Table 1 is the minimum immutable bill-of-materials (BOM) diff. Populate observed values, not expected defaults.

The table turns “same model” into a falsifiable statement. Exact dependency pinning matters because a vLLM artifact is only one layer of the serving environment. Pip's secure-install guidance requires exact version, URL, or path pins in hash-checking mode ^[27].

Determine the runner that actually executes

Create one qualification record per workload, not one per cluster. Each record should include:

• Identity: model and tokenizer or processor commit, served name, trust-remote-code state.

• Backend: CUDA, ROCm, XPU, or CPU artifact and resolved digest.

• Execution: effective MRV1 or MRV2 runner, attention backend, eager or graph mode.

• Precision: weight quantization, activation and KV-cache data types.

• Parallelism: tensor, pipeline, data, expert, and sequence settings.

• Advanced paths: speculative method and draft model, LoRA adapter, multimodal processor, KV connector.

• Evidence: startup log, effective configuration dump, route inventory, test case IDs, and results.

Start the candidate with production-equivalent flags, then capture the effective configuration after feature validation. Assert the runner in logs or instrumentation, exercise each optional feature, and deliberately test unsupported combinations in staging. A startup success with an MRV1 fallback is not an MRV2 qualification. Conversely, forcing MRV2 does not prove that every feature is implemented correctly.

Build a correctness regression suite

Correctness acceptance should compare application-visible invariants before considering performance:

• Request validation: valid and invalid schemas, missing fields, bounds, media types, and status codes.

• Tokenization: rendered prompts, control tokens, token IDs, truncation, and special-token handling.

• Generation: stop strings, stop token IDs, finish reasons, streaming chunks, usage accounting.

• Probabilities: sampled token, returned logprob, top alternatives, and absent or null fields.

• Structured output: schema validation with required fields, enums, arrays, nested objects, and refusals.

• Tool calls: names, arguments, IDs, parallel calls, parser failures, and streamed deltas.

• Multimodal: image, audio, or video placement, processor revision, modality limits, and malformed inputs.

• LoRA: static adapter selection, adapter identity, concurrency, and base-model fallback.

• Failure behavior: overload, cancellation, timeout, invalid adapter, oversized request, and dependency failure.

Include static LoRA selection and multimodal content placement in the exact request fixtures. Multimodal template state can live on the processor rather than the tokenizer, which makes the processor revision part of reproducibility ^[21]. These are examples of semantic contracts that token-only comparisons can miss.

Classify differences before failing them. A schema violation, changed stop behavior, wrong tool call, or new authorization result is application-visible. Small floating-point or probability differences may be acceptable if outputs remain within a predeclared semantic and statistical envelope. Record the seed, but do not make bitwise identity the only criterion.

Artifact, Capacity, and Rollout Strategy

Select and pin the correct artifact

Match artifact family to the host rather than trying the default wheel everywhere. NVIDIA states that the installed driver must meet or exceed the CUDA Toolkit minimum, and documents driver 580 or later for CUDA 13.x minor-version compatibility ^{[28] [29]}. Its table lists CUDA 12.9 GA minima of 575.51.03 on Linux x86_64 and 576.02 on Windows x86_64 ^[30]. AMD likewise directs operators to a release-specific matrix covering ROCm, operating systems, and accelerators ^[31].

For each candidate host:

• Resolve: convert the tag to registry index and platform-manifest digests.

• Inspect: capture OS, architecture, CUDA or ROCm libraries, Python, PyTorch, Triton, FlashInfer, and vLLM commit.

• Verify: compare driver, firmware, runtime, and GPU support against the relevant matrix.

• Lock: hash wheels and every Python dependency; retain the lockfile with the test result.

• Archive: cache the previous image, configuration, model manifest, proxy policy, and dashboards.

• Label: attach the candidate digest and configuration hash to metrics, logs, and traces.

Docker's inspection tooling exposes image and manifest-list digests for this record ^[32]. Do not infer a runtime from a floating tag or generic installation page.

Qualify capacity with the production distribution

Performance testing should preserve fixed model revisions, hardware, precision, input and output length distributions, concurrency or request-rate shape, and feature mix. Warm and cold behaviors are separate tests because image pull, model load, kernel compilation, graph capture, and cache population affect readiness.

Measure at least:

• Startup: image pull, process start, model load, graph capture, warmup, and healthy-ready time.

• Latency: TTFT, ITL, time per output token, and end-to-end p50, p95, and p99.

• Capacity: completed tokens per second, requests per second, maximum admitted concurrency, and queue time.

• Reliability: HTTP and engine error rate, cancellation, timeout, crash, GPU reset, and out-of-memory count.

• Memory: model footprint, free memory before load, KV-cache allocation, KV utilization, and high-water mark.

• Quality: structured-output pass rate, tool-call pass rate, stop correctness, and task-specific scoring.

Prometheus histograms permit latency percentiles to be aggregated across replicas with histogram_quantile() ^[7]. Check the candidate's live metrics before changing dashboards. NVIDIA similarly recommends using the exporter's /metrics endpoint to see effective runtime output ^[33].

Canary and rollback

Run the old and candidate artifacts on identical qualified hardware. Replay a permitted trace or shadow sanitized traffic, then route a small live share only after correctness and exposure gates pass. Kubernetes defines a canary as a new version receiving a small percentage of production traffic alongside the stable version ^[34]. Its Gateway API example shows an explicit 90/10 stable-to-canary split ^[35]. That is an example, not a universal starting percentage.

Define abort thresholds before traffic starts. Include correctness mismatch, unauthorized route reachability, crash or reset, out-of-memory event, tail-latency breach, error-rate breach, loss of rollback capacity, and rollback-time breach. On abort, remove candidate traffic, restore stable capacity, and preserve evidence. Kubernetes notes that scaling a canary to zero retains its configuration for inspection ^[36].

Endpoint Exposure and Security Acceptance

Treat the live route table as the source of truth

The acceptance test must probe the deployed middleware with and without credentials.

Table 2 turns the documented exposure classes into a proxy policy. Exact enabled routes depend on flags, task, and plugins, so export app.routes or an equivalent live inventory.

The important distinction is reachable versus enabled. The security guide says /tokenizer_info can expose chat templates and tokenizer configuration ^[37]. The proxy should still deny optional development and profiler paths even when startup policy says they are disabled. OWASP independently recommends keeping management endpoints off the public Internet ^[38].

Enforce a narrow production boundary

The acceptance sequence should be mechanical:

• Enumerate: capture every live method and path after all plugins and routers load.

• Classify: assign end-user, service, operator, monitoring, or no audience.

• Deny: block direct network access to vLLM and default-deny every unapproved path at the proxy.

• Authenticate: enforce workload or user identity at the proxy, including on nominally protected vLLM routes.

• Authorize: separate inference, monitoring, and model-changing administration roles.

• Constrain: limit body size, tokens, output count, concurrency, request rate, and time.

• Observe: log principal, route, model, status, latency, admitted resource envelope, and policy decision.

• Test: send authenticated and unauthenticated requests to every route from allowed and disallowed networks.

OWASP recommends maximum sizes for incoming parameters and payloads, plus per-client interaction limits ^{[39] [40]}. It also advises keeping management endpoints off the public Internet and recording audit events before and after security-relevant actions ^{[38] [41]}. NIST's zero-trust guidance reinforces that network location or ownership alone does not create implicit trust ^[9].

Resource controls need workload-specific values. The vLLM guide documents VLLM_MAX_N_SEQUENCES with a default of 16,384, but that ceiling is not a safe production quota for every model or GPU ^[42]. Set smaller request and aggregate limits from measured memory and latency envelopes. NGINX can restrict access by client address, but address rules should complement identity-aware authorization, not replace it ^[43].

Data Analysis and Evidence

The quantitative core of an upgrade is a paired experiment, not a borrowed benchmark. Old and new candidates must receive the same trace distribution on the same qualified hardware, with the same model, tokenizer, precision, prompt-length distribution, output-length distribution, request rate, concurrency policy, and feature mix. Preserve raw request IDs and metric exports so that aggregate changes can be traced to individual failures.

Table 3 is a results and abort matrix. Operators must fill the threshold column from their service-level objectives and risk policy. This report intentionally does not invent universal limits.

The v0.29.0 benchmark can cap actual concurrency separately from arrival rate ^[44]. Use that separation to test both queueing and execution saturation. For KV capacity, measure free KV bytes after model load and graph capture, then estimate the per-request envelope from the production token distribution:

admitted concurrency <= floor((measured free KV bytes - reserve bytes) / measured p95 KV bytes per request)

This is a planning bound, not a theoretical constant. Validate it by load testing because prefix caching, multimodal state, speculative decoding, parallelism, and fragmentation can change realized memory. vLLM permits an explicit KV-cache byte allocation for finer control and reports group-aware cache capacity per data-parallel engine ^{[45] [46]}.

Version labels are useful only when bounded. Prometheus notes that every unique label-set combination creates a new time series, so use controlled values such as release, artifact digest prefix, runner, and configuration ID rather than request-level identifiers ^[47]. The evidence pack should contain the BOM diff, qualification matrix, raw benchmark output, route audit, proxy policy, dashboards, abort decisions, rollback timing, and named approvers.

Implications and Future Directions

MRV2's default status changes the baseline for future upgrades. MRV1-specific optimization is a shrinking investment, but the immediate response should not be to force every workload onto MRV2. Instead, classify remaining V1 workloads, document why each falls back, and retest them against subsequent patch releases. This makes retirement work visible without turning an unsupported combination into an availability risk.

Three practices follow from the evidence:

• Make runner identity observable. Emit release, artifact digest, runner, backend, model revision, and configuration hash as bounded deployment metadata.

• Promote feature paths, not images. A digest is eligible only for the exact model and option combinations that passed.

• Treat routes as deployable state. Re-audit route inventory whenever vLLM, flags, plugins, tool servers, or proxy configuration changes.

• Retain evidence by patch. Each 0.29.x candidate should have a new BOM diff, targeted regression run, exposure audit, and rollback rehearsal.

• Separate compatibility from optimization. First prove application contracts, then tune graph capture, KV allocation, batching, and concurrency.

• Recheck metrics before dashboard migration. Metric names and labels are code-version behavior, not timeless API contracts.

Optional tool-server and plugin surfaces should be part of configuration review and route testing, not assumed absent forever. Apply the same bounded-input and per-client request controls to these paths that OWASP recommends for API resource consumption ^[40].

The durable outcome is a repeatable promotion system. It can qualify later vLLM patches, new GPUs, new model revisions, and new attention or speculative paths using the same evidence structure. That is more valuable than a one-time claim that 0.29 is faster or safer.

Frequently Asked Questions (FAQs)

Are there vLLM 0.29 breaking changes?

It is an execution-path change with possible application-visible effects, even where HTTP schemas remain compatible. MRV2 becomes the default, unsupported combinations can fall back to MRV1, and tokenizer, structured-output, tool-call, memory, and graph behavior require regression tests. Treat it as a controlled production change rather than assuming semantic compatibility from the version number.

How to validate a vLLM upgrade

Record the complete workload tuple, capture the effective runner after startup validation, and exercise every enabled feature. Compare old and new correctness first, then capacity and latency under an identical trace distribution. The test result must say which runner actually executed, not merely which default the release announced.

Does --api-key secure every vLLM endpoint?

No. Versioned documentation lists unauthenticated inference variants, operational controls, utilities, and optional development surfaces. Put vLLM behind a reverse proxy, deny direct access, allowlist exact routes and methods, apply separate authorization for administration, and test every live route without credentials. OWASP's guidance to keep management endpoints off the Internet supports that design ^[38].

Which vLLM 0.29 artifact should be installed?

Choose the release family that matches the qualified GPU backend and host driver, then pin the resolved digest or wheel hash. The default PyPI wheel should not be assumed suitable for every CUDA, ROCm, or XPU host. Preserve the installed package inventory and runtime inspection with the test evidence.

What should trigger rollback?

Operator-defined triggers should include critical output-contract mismatch, unauthorized route reachability, crash or GPU reset, out-of-memory, tail-latency or error-rate breach, and inability to restore the prior qualified capacity within the recovery objective. Kubernetes supports rollback to a retained specific revision, but that mechanism still needs a timed rehearsal ^[48].

Conclusion

vLLM 0.29.0 should be promoted by evidence, not by version label. Its September 2026 release moves all models to an MRV2 default while preserving conditional MRV1 paths, and it publishes distinct CUDA, ROCm, XPU, and CPU artifacts. That combination requires an immutable BOM and a feature-path qualification matrix for every production workload.

The acceptance order matters. First, prove request validation, tokenization, chat templates, stop conditions, logprobs, structured output, tool calls, multimodal processing, LoRA, and failure behavior. Second, measure startup, KV capacity, concurrency, latency distributions, throughput, error rates, and GPU memory using the real workload shape. Third, enumerate the deployed routes and prove that the reverse proxy denies every path, method, identity, and network combination outside the approved policy.

Promotion should proceed through a version-labeled canary with predeclared abort thresholds and a timed rollback. The retained evidence pack should include artifact hashes, dependency locks, runtime inspection, effective runner, raw regression data, metrics mapping, route audit, proxy policy, and rollback timings. With those controls, 0.29 becomes a bounded engineering change. Without them, the new default runner and incomplete built-in authentication boundary remain assumptions hidden behind a successful health check.