vLLM CUDA Graph Memory Budget: Capture Sizes and KV Headroom

Executive Summary

For an existing NVIDIA allocation, the vLLM CUDA graph memory question is not "how many bytes does a graph cost" but "which graph modes and capture shapes can be qualified without shrinking KV (key-value) cache below what the latency objective needs." Current vLLM documentation states that CUDA graphs are on by default and "take up extra memory in the GPU" ^[1], that the default mode is FULL_AND_PIECEWISE, which "requires the most memory and takes the longest to capture" ^[2], and that capture sizes can be reduced or graphs disabled with the eager flag. Those pages give no per-graph byte figures, so the number has to be measured on the reader's own model, backend, parallelism and revision.

Three facts shape the measurement plan. First, vLLM runs a dummy profiling forward pass, then a separate graph-memory estimation pass, and only then sizes KV cache ^[3]; graph memory estimation has been on by default since v0.21.0 ^[4]. Second, the estimate is not the actual: in the vLLM team's September 2026 GB300 experiment, one TP4DP4+MTP row reserved 7.56 GiB for graphs and used 9.90 GiB, while other rows reserved more than they used ^[5]. That is a scoped vendor observation, not a reserve rule. Third, the PyTorch allocator sees only its own allocations, so memory held by libraries such as NCCL appears only in device-wide counters ^[6].

The recommended method is a lifecycle ledger: per rank, record device-used, allocated and reserved memory at fixed stages (context, weights, profile run, graph estimate, capture, KV initialization, warm steady state, extreme requests), one fresh process per cell and one changed variable per cell. Run two tracks and label them separately: a fixed explicit KV budget, which skips profiling and the graph estimate and isolates execution overhead, and automatic KV allocation, which exposes the full deployment trade-off.

Selection should use observed shape coverage (eligible steps served by a captured graph divided by eligible steps), padding evidence from vLLM's graph metrics, and goodput under the reader's own TTFT (time to first token) and inter-token latency limits. Published cross-runtime evidence is directional only: TensorRT-LLM measured about 10 MB per additional graph on one DeepSeek-R1 configuration ^[7], yet a 17.2% KV capacity reduction on another model (DeepSeek-V3.2, where 210 additional graphs beyond a +64 configuration were captured) ^[7]. Cold capture time, rolling-restart overlap and steady-state memory are separate costs. No GPU benchmark was produced for this report; all figures are quoted from named sources and scoped to their stated conditions.

Introduction and Background

Capacity engineers who own an NVIDIA inference platform often face the same sequence: a launch-overhead diagnosis has already justified testing CUDA graphs, the model and GPU allocation are fixed, and the open question is which graph modes and capture shapes to qualify. This report treats that as a measured execution-memory tuning decision, not a weights or bytes-per-token calculation. GPU Smith, an independent private AI infrastructure engineering consultancy, frames the work as a reproducible worksheet: the output is a measured non-KV budget that a KV capacity plan can consume, together with evidence for or against buying more HBM (high-bandwidth memory).

A CUDA graph records a sequence of GPU operations once and replays it with a single launch. NVIDIA documents that CPU launch cost is reduced because "much of the setup is done in advance" ^[8], and PyTorch documents that a replay skips argument setup and dispatch layers in Python, C++ and the CUDA driver ^[9]. The benefit is largest when host overhead exceeds GPU run time, which PyTorch's engineers describe as the small-batch regime ^[10]. A preprint measuring batch-1 decode found a 1.259x latency improvement on an H100 at 2,048 context but only 1.028x on an L4 ^[11], which illustrates why the diagnostic comes first and why gains are hardware specific.

The memory cost follows from replay mechanics. A captured graph acts on the same virtual addresses every time it replays ^[9], so static buffers and a graph-private allocator pool must persist ^[12]. Capture-time allocations come from that private pool, which "stays alive" until the graph and its tensors go out of scope ^[9]. By default each graph gets its own pool, which PyTorch notes "sometimes needlessly wastes memory" ^[9]; pools may be shared only if graphs replay in capture order ^[9].

Scope and dating matter. The vLLM documentation pages read for this report are the "latest" builds and carry no version number, and source files were read from the main branch on 2 November 2026 without a recorded commit. Behavior below is therefore a checklist of what to verify in a pinned deployment, not a claim about any released version. The PyTorch 2.8 notes used here are an older pinned reference, not an assertion about the deployed PyTorch version.

Key Changes: Graph Modes, Eligibility and Capture Shapes

Change 1: Modes are separate from compilation and can be downgraded

The current design separates graph behavior from compilation: the redesign "made full CUDA Graphs support orthogonal to compilation" ^[2]; full graphs work with or without compilation. Five modes exist. Table 1 below summarizes what each captures and where its documented memory, startup or fallback implications sit.

The table shows why "requested mode" and "effective mode" are different ledger columns. The design document states that not all modes work with every attention backend and that vLLM downgrades to the closest supported mode ^[2]; a backend that supports graphs only for uniform decode batches turns a FULL request into FULL_AND_PIECEWISE or FULL_DECODE_ONLY ^[2]. For hybrid models the minimum capability across all backends decides the model's capability. A run whose startup log shows a different effective mode than the configuration file is a different experiment and must be recorded as such.

Two further eligibility rules affect the sweep. Attention-fusion custom passes that need the whole graph make vLLM disable piecewise compilation and use FULL-style modes. And at runtime, a central dispatcher is the "single source of truth" for which graph keys exist; it searches keys in priority order and, if no key matches, returns NONE, meaning eager execution ^[2]. Shape coverage is therefore a dispatch outcome that must be observed, not inferred from the capture list.

Change 2: Default capture sizes are derived and capped

Capture-size defaults are generated, not fixed. When no list is given, sizes are inferred from the config; the source describes 1, 2 and 4, then multiples of 8 up to 256, then multiples of 16 ^[14]. The ceiling defaults to 512, or 1024 on data center Blackwell GPUs, to avoid out-of-memory (OOM) in tight-memory cases, and the same text notes that larger graphs increase startup time and memory ^[13]. A user-supplied list overrides the derivation ^[14], and a graph captured for one size can only serve that size, so batches between captured sizes are padded upward, which the design notes as a padded token length in the dispatch key.

One documentation discrepancy deserves recording. The conserving-memory example comments that the default list "goes up to max_num_seqs" ^[1], while the source docstring and configuration reference describe a cap of 512 or 1024. The two statements are compatible only when max_num_seqs is below the cap, so the ledger should store the actual captured list printed at startup. The same engine-argument reference says max_num_seqs "sizes the model runner" and graph capture ^[15], and a separate max_num_active_seqs keeps batches smaller "without shrinking runner or graph capacity". That is the answer to the max_num_seqs and CUDA graph memory question: lowering it can shrink both runner buffers and the capture set, but also lowers admitted concurrency, so it is a workload decision before it is a memory one.

A field example shows the other direction. In an August 2026 vLLM post the default cap of 512 covered concurrency up to 3,072 for a decode endpoint, and the authors raised the cap for higher concurrency while stating they were "not certain" the change was required ^[16]. Treat that as a reminder to test whether a larger set earns its memory, not as a recommendation.

Change 3: Feature paths add graph variants

Several features change which graphs exist. For speculative decoding, uniform decode batches have query length 1 plus the number of speculative tokens, as in the validation pass ^[2], and many backends support full graphs only for uniform batches. The September 2026 vLLM experiment used MTP (multi-token prediction) with 3 speculative tokens ^[5]. Its authors advise treating the MTP concurrency rows as an upper bound rather than reachable concurrency ^[5], a caution that applies to any derived-capacity column.

For LoRA (low-rank adaptation) serving, vLLM by default creates separate graphs for cases with and without active LoRA, at the cost of startup time and slightly higher memory ^[17]. Multimodal encoders have their own opt-in graphs ^[13] that are orthogonal to decoder graphs. For data parallel (DP) serving, padding to the maximum token count across ranks applies when graphs are enabled ^[18], and it is enabled only when the synchronized mode across ranks is not NONE. Each of these is a refresh trigger in the checklist later in this report.

Key Changes: Memory Estimation and KV Sizing

Change 4: Graph memory is estimated before KV cache is sized

The startup order explains the measurement stages:

• Profile run: the worker profiles peak model memory to decide how much the KV cache can use ^[3], by executing a forward pass with dummy inputs ^[3].
• Graph estimate: it then profiles graph memory if graphs will be captured ^[3], using a temporary pool to avoid fragmenting the main pool ^[19].
• KV sizing: the cache is sized from what the budget leaves after those two measurements.
• Warmup and capture: kernel warmup precedes capture ^[3], and real capture goes large shapes first so smaller ones reuse the pool ^[19].

The change that made this default was a pull request merged on 21 April 2026 that enabled graph memory profiling and raised the default gpu_memory_utilization to 0.92 to compensate ^[20]; the source documents it as enabled by default as of v0.21.0, and a later pull request extended it to Model Runner V2 ^[21]. If the estimate is disabled, the code warns that graph memory is "not accounted for" in KV allocation ^[3].

The September 2026 vLLM post describes the mechanism in its own words. Because graphs use static addresses, vLLM is "forced to use that estimate to claim the whole amount beforehand" ^[5]; it estimates the largest graph and assumes later graphs add a similar extra increment ^[5], because the second-largest graph still costs some extra memory. The consequence is direct: "KV cache size depends directly on how good that estimate is" ^[5]. The authors suggest estimating graph size manually when optimizing final KV percentages ^[5].

Two accounting details matter for the ledger. The profiling context splits GPU memory into other processes, torch memory and non-torch memory in the instance ^[22], and measures total consumption with mem_get_info rather than allocator counters ^[22], adding only transient peak headroom so persistent allocations are not double counted ^[22]. The activation pass runs in eager mode, because real batches may exceed the largest graph and run eagerly ^[5].

Change 5: An explicit KV budget skips profiling

The kv_cache_memory_bytes setting defaults to None, meaning vLLM infers the size. When it is set, the documented behavior has four parts:

• Profiling is skipped: passing the logged value back on the next boot skips both memory profiling and the graph estimation pass ^[23].
• Utilization is ignored: the source logs that profiling was skipped and that the setting does not respect gpu_memory_utilization ^[3].
• Errors move: a conservative value "caps batch concurrency (and therefore throughput)" ^[23], while an optimistic one fails at allocation time.
• Portability is limited: the value is valid only on the same GPU with the same initial free memory ^[23].

A separate extensible-KV option sizes the cache from memory free after graph capture rather than a profiling estimate ^[15], and the documented default utilization changes to 1.0 with it. Whether that option is available or enabled in a pinned deployment is a fact to verify.

Change 6: The allocator does not see everything

The utilization fraction is applied to total device memory: the request is computed from total memory and validated against free memory ^[24], and the setting is per instance. Inside a process, PyTorch's snapshot tools "only have visibility" into memory obtained through its allocator ^[6], memory allocated directly from CUDA APIs is not seen ^[6], and NCCL is the documented example ^[6]. memory_allocated counts tensor-occupied memory ^[25], memory_reserved counts what the caching allocator manages ^[26], and mem_get_info reports device-global free and total memory ^[27]. The consequence for graph memory is that the allocator can account for a private pool while device-wide usage rises by more, with the residual attributable to libraries and metadata.

Implementation Considerations and Process Changes: The Lifecycle Worksheet

Step 1: Fix the experiment conditions

Before any cell runs, freeze and record four groups of inputs:

• Model and runtime: weights, tokenizer, precision, and the vLLM and PyTorch revisions.
• Hardware stack: GPU, driver, CUDA version, attention backend and effective runner.
• Parallelism and features: tensor parallel (TP) and DP layout, speculative decoding and LoRA settings.
• Traffic and targets: the traffic profile, the quality gate, and the TTFT and inter-token latency (ITL) requirements.

The metric definitions must also be fixed, since vLLM's documentation warns that "metric terminology is not standardized" across tools ^[28]. ITL approximates time per output token (TPOT) only when each streamed output holds one token ^[29]; with speculative decoding a streamed output can bundle several tokens, so ITL and TPOT diverge and a latency objective written in ITL and one written in TPOT are different gates.

Step 2: Read what is actually captured

For every cell, store four items:

• Mode: requested and effective, including any downgrade message.
• Sizes: the captured size list printed at startup.
• Backend: the graph capability of each attention backend in use.
• Dispatch: padded and unpadded token counts and runtime dispatch modes, from the graph metrics option ^[15].

Separate compilation time from capture time and from replay behavior, since the two features are orthogonal.

Step 3: Measure memory across the lifecycle

Table 2 below is the blank ledger; cells marked "(fill)" are left empty for the reader to complete. One copy is filled per rank and per cell; device-used is total minus free from mem_get_info, and the last column keeps the non-torch residual explicit.

The ledger works only if windows do not overlap. Reserved memory is the sum of allocator segments ^[6], and allocated memory is a subset of it, so adding allocated to reserved double counts. Subtracting a peak from one window from a baseline from another is equally misleading; take stage deltas from the same process and clear peaks between stages. The NVIDIA troubleshooting guide notes that even one graph implies two pools, global and graph-private, whose fragments cannot be shared ^[12], and that each additional pool adds an isolated region ^[12]. Freed tensors inside capture remain reserved but unusable ^[12], which is why a graph delta in memory_reserved can exceed what the graph "needs".

Define the quantities explicitly:

• Graph estimate error E = reported graph reservation estimate minus measured graph-pool quantity, computed only when both cover the same rank, window and pool definition; otherwise leave E blank.
• Capture residual = device-used delta across capture minus reserved delta across capture; attribute it to non-torch memory only after confirming no unrelated process changed.
• Observed captured-shape coverage = eligible observed steps served by a captured graph divided by eligible observed steps, with padding evidence from the dispatcher metrics.
• Padding waste = padded tokens minus unpadded tokens, divided by padded tokens, from the same metrics.
• Goodput = completed requests per second meeting the stated service-level objectives ^[32].

NVIDIA also documents a graph memory footprint query for graph memory nodes, with reserved and used attributes ^[8] and a high watermark that can be reset ^[33]. Those attributes describe graph memory nodes owned by the driver ^[8]; whether they cover PyTorch private pools in a given deployment must be verified before they are used as a cross-check.

Step 4: Sweep a small informed set

Keep the matrix small and run every cell in a fresh process, changing one variable at a time, with identical inputs and enough repeats to see local variability. A defensible first set is:

• Baseline production captures: the effective mode and the full captured list from the current deployment.
• Reduced shape set: captures concentrated where the shape histogram says traffic lives, with the maximum covering the admitted batch.
• Justified alternate mode: for example FULL_DECODE_ONLY for decode-dominant instances, where the docs say it saves the memory PIECEWISE graphs need.
• Eager control: graphs disabled, to measure what the production set buys and costs.

Run each cell on two tracks, labeled separately. Track A (fixed explicit KV budget) holds KV capacity constant so differences are execution overhead, with the profiling and estimate steps skipped; verify what else the setting overrides in the pinned release. Track B (automatic KV allocation) lets each cell size KV itself, so graph memory shows up as lost KV, which is the full deployment trade-off. Comparing eager and graph cells on Track B without recording the KV each received would confuse the two effects; the vLLM post shows why, since its KV capacity depends on the estimate.

Step 5: Choose the capture set and plan rollback

Choose from coverage, padding and SLO-qualified goodput, not from the lowest memory figure. PyTorch's compiler guide says re-recording is profitable when shapes are limited, such as inference batch sizes ^[34], and suggests padding to a few fixed shapes when shapes vary ^[34]. Before promotion, require four checks:

• OOM test: the extreme cases from S7 complete without allocation failure.
• Fallback test: observed coverage and the eager share meet the target, with the dispatcher evidence attached.
• Correctness test: outputs match the baseline cell on the agreed quality set.
• Rollback: the previous configuration stays deployable.

Treat three costs separately:

• Cold capture time: a preprint states capture "often dominates startup" ^[35], and TensorRT-LLM found capture time scales linearly with graph count there ^[7]. vLLM's optimization levels explicitly trade startup time for performance, and a release note describes opt-in reuse of the memory-profiling result across boots ^[36].
• Rolling-restart overlap: when two instances share a GPU during a restart, each instance's utilization setting is independent, so the overlap window needs its own memory check.
• Steady-state memory: stages S6 and S7 of the ledger, which decide whether the chosen set survives production traffic.

Data Analysis and Evidence

Published estimate versus actual graph memory

The only first-party vLLM table found that separates the graph estimate from actual use is the September 2026 GB300 post. Its conditions are specific: measurements ran on a GB300 cluster connected via NVLink72 ^[5], with the default 8% reserve of a 276.62 GiB baseline ^[5], and graph rows labeled by topology and MTP setting. The measurement runs used aggregated decode topologies with the FULL_DECODE_ONLY mode ^[5]. The authors state the estimate "often exceeds" actual need "by a wide margin" ^[5], and one row shows the opposite ^[5]. Table 3 reproduces the rows as read from the post's table (the fragment links point to the adjacent prose, since table cells are not addressable text), and computes estimate error as estimate minus actual.

The spread, from a 24% shortfall to a 56% surplus, is the reason a fixed percentage reserve is an unsafe general rule: sign and size changed with topology, speculation and graph count in a single experiment. For scale only, the post's default 8% reserve of 276.62 GiB is about 22.1 GiB, so the largest shortfall of 2.34 GiB was roughly a tenth of that reserve; this arithmetic describes one fleet and sets no reserve for another. The post also reports that memory outside weights and KV cache comes to about 20 GiB in every case it measured, and it calls out that results are conditioned on its hardware and software. Estimate and actual may not share a window or definition, so the E column is informative only to the extent the post's definitions align with the reader's logs.

Cross-runtime evidence on shape sets

Other runtimes show the same tension, with different numbers. SGLang advises reserving memory for activations and graph buffers when raising its static memory fraction ^[37], enables graphs by default only for small batch sizes ^[37], and warns that larger graph sets consume more memory, so the static fraction may need lowering ^[37]. Its static fraction covers weights and the KV pool only ^[38]. TensorRT-LLM pads an incoming batch to the next captured size ^[7], and its older default captured 23 sizes in a roughly doubling progression ^[7], which could cause up to 50% padding waste at high concurrency ^[7]. It measured about 10 MB per extra graph on DeepSeek-R1 with TP=4 ^[7], and found a larger graph footprint cut KV token capacity by 17.2% on DeepSeek-V3.2 against 8.6% on R1 ^[7]. The lesson is not a transferable number; it is that per-graph cost, padding waste and KV loss depend on the model, so each must be measured. PyTorch's own compiler documentation adds that a separate pool for graph allocations can raise a program's memory ^[34] and that per-launch metadata can be significant when graphs are re-recorded ^[34].

Latency evidence

Latency benefits are conditional. The batch-1 preprint above found the benefit shrank from 1.259x to 1.028x between GPUs ^[11], and a separate cold-start study of vLLM describes startup as predominantly CPU bound ^[39]. Latency and throughput trade-offs therefore must come from the reader's service-level objective. vLLM's benchmark CLI accepts goodput objectives as key-value pairs in milliseconds ^[40]. A smaller batched-token limit also improves ITL because fewer prefills slow decodes ^[23], a confounder to hold constant across cells.

Cost per accepted output

Cost should be computed from observed accepted output and reader-supplied GPU-hour prices, never from a graph delta divided by theoretical KV bytes. The formula is: cost per million accepted output tokens = (GPU count × price per GPU-hour) ÷ (accepted output tokens per hour ÷ 1,000,000). (Hypothetical Example) Suppose cell A (production captures) sustains 36 million SLO-passing output tokens per hour on 8 GPUs and cell B (reduced captures) sustains 33 million, with a reader price of $p per GPU-hour. Cost per million tokens is 8p ÷ 36 = 0.222p for A and 8p ÷ 33 = 0.242p for B, so B would need its freed memory to buy more than 9% extra qualified output to win. These numbers are illustrative, not measurements.

Implications and Future Directions

Capacity decisions. A shortfall justifies more HBM only after the reader has shown that a qualified smaller capture set, or a justified mode change, still leaves KV below what the latency objective needs. For scale, a 2023 paper reported about 65% of a 40 GB A100's memory going to weights and close to 30% to dynamic KV state for a 13B model ^[41], which shows how thin the non-KV margin can be on small-memory parts; NVIDIA describes the H200 as the first GPU with 141 GB of HBM3e ^[42]. These are context, not purchase advice, and the older paper's setup differs from current serving stacks.

Hypothesis, unproven here. Smaller, coverage-driven capture sets may defer some HBM purchases, but this research produced no fleet measurement. The worksheet exists to test exactly that claim per workload.

Where the ground is moving. Graph memory estimation has changed defaults within 2026, including the utilization default ^[20], a Model Runner V2 extension ^[21] and a release-note item on lower memory for capturing large graph sizes ^[36]. Other vendors document similar mechanisms: AMD's HIP graphs documentation describes graph memory pools and notes graphs pay off only for workloads that replay many iterations ^[43]. Refresh the ledger, not the conclusion, when any of these inputs change.

Refresh checklist. Re-run the affected cells whenever any of the following changes:

• vLLM, PyTorch, CUDA or driver revision: estimate logic, defaults and allocator behavior move between releases.
• Attention backend or effective runner: eligibility and fallback change which graphs exist.
• Model or quantization: per-graph activation size and weights change.
• Speculative decoding settings: uniform decode query length changes.
• LoRA or multimodal enablement: added graph variants.
• TP, DP or expert layout: per-rank weights, padding across ranks and communication buffers change ^[44].
• Traffic shape histogram: coverage and padding waste shift with the workload.

Frequently Asked Questions (FAQs)

How much memory do vLLM CUDA graphs use? The documentation says only that they take up extra memory. Published numbers are scoped: one vendor experiment shows 0.59 to 9.90 GiB depending on topology and graph count (Table 3), and another runtime measured about 10 MB per additional graph in one configuration. Measure your own.

Which capture sizes does vLLM use, and can they be changed? By default sizes are inferred from the config; setting cudagraph_capture_sizes captures only the specified sizes ^[45], and the conserving-memory guide shows a reduced list.

How does gpu_memory_utilization interact with graphs? The fraction is applied to total device memory and is per instance; with graph estimation on, the estimate is part of the budget before KV is sized. Raising the fraction gives KV more space ^[23] but also removes the cushion that absorbs estimate error.

How do graphs change KV memory headroom? With automatic sizing, the graph estimate is deducted before KV is sized, so "KV cache size depends directly on how good that estimate is" ^[5]. A larger or denser capture set can therefore reduce KV capacity; read the engine's available-KV figure per cell on Track B instead of converting a graph delta into concurrency.

Does --enforce-eager free more KV cache? It skips compilation and capture for the fastest startup. How much KV you gain depends on the estimate and the run's accounting, so compare eager and graph cells on Track B and record the KV each received.

Why does CUDA OOM appear after startup or at capture? NVIDIA lists seven main causes of OOM with CUDA graphs ^[12], including persistent static inputs ^[12] and cached memory that cannot be released during capture ^[12]. Late-loaded modules can also fail if VRAM was fully claimed at startup ^[30]. Use the ledger to see which stage moved.

What does max_num_seqs do to graph memory? It sizes runner buffers and graph capture, so lowering it shrinks both but caps concurrency.

What is the latency and throughput trade-off? Graphs reduce host launch overhead, with gains that depend on GPU and batch size ^[11]; the cost is memory and startup time. Decide with SLO-qualified goodput ^[32].

Conclusion

The CUDA graph memory decision on a fixed allocation is an empirical one. vLLM's documentation defines the modes, fallbacks and size derivation clearly enough to design a sweep, but it does not supply a per-graph byte cost, and the one first-party table that compares graph estimates with actual use shows estimates that are both above and below actuals within a single experiment. The defensible response is a lifecycle ledger: record requested and effective mode, captured sizes, observed coverage and padding, then memory per rank at fixed stages in fresh processes, with a fixed-KV track for execution cost and an automatic-KV track for deployment cost.

Qualify a capture set by SLO-qualified goodput and cost per accepted output, test OOM, fallback and correctness paths including extreme requests, and keep a rollback. Treat cold capture time, restart overlap and steady-state memory as separate line items. Share the resulting non-KV budget with the KV planning process, and rerun the affected cells whenever the runtime, backend, model, speculation, LoRA or parallelism changes. Until a cell is measured, any fixed percentage reserve is a guess.