Prefix Caching and KV Cache Offload to CPU RAM and NVMe

Why prefix reuse matters

Chat, RAG and agent workloads keep resending the same long prefix: a system prompt, tool definitions, a retrieved document, the conversation so far. Without reuse, every turn pays full prefill on tokens the server has already processed. Prefix caching keeps the key/value (KV) tensors for those tokens and skips recomputing them. Offload extends that cache from HBM to CPU DRAM, local NVMe or a remote store, so more prefixes survive long enough to be hit again.

The short answer: reuse from HBM is nearly free, and fetching from CPU DRAM over PCIe Gen5 beats recomputing for almost any prefix worth caching. NVMe and remote tiers win only above a minimum prefix length, which depends on the model, the GPU and the storage path. On large GQA models that minimum is low. On small models and fast GPUs it can sit near 8k tokens or higher. The arithmetic below uses a named model so you can redo it for yours.

Prefix caching only speeds up prefill. vLLM's documentation says APC "only reduces the time of processing the queries (the prefilling phase)" and "does not reduce the time of generating new tokens (the decoding phase)" ^[1].

How automatic prefix caching works

vLLM. Automatic Prefix Caching (APC) "caches the KV cache of existing queries", so a new query sharing a prefix can skip the shared part. It is enabled with enable_prefix_caching=True ^[1]. Under the hood, vLLM hashes each KV block from the parent block's hash, the block's tokens and extra keys such as LoRA IDs, multimodal input hashes and cache salts. It notes "We only cache full blocks", and freed blocks are evicted least-recently-used first. The default hash is sha256; the faster xxhash option is flagged as not cryptographically secure ^[2]. Because each block's hash includes its parent's, a block only matches when every token before it also matches.

SGLang. RadixAttention stores cached prefixes in a radix tree, so the longest shared prefix across many requests is found and reused automatically. The SGLang paper reports "up to 6.4x higher throughput compared to state-of-the-art inference systems" ^[3]. That figure comes from the whole runtime on the paper's structured-program benchmarks, not from RadixAttention alone.

For operators the two behave alike: exact-prefix reuse, eviction under pressure, nothing kept beyond HBM without a lower tier. Your traffic's hit rate matters more than the engine.

KV bytes per token: a worked example for Qwen2.5-72B

The Qwen2.5-72B-Instruct config.json gives num_hidden_layers: 80, num_key_value_heads: 8, num_attention_heads: 64, hidden_size: 8192 and torch_dtype: bfloat16. Head dimension is 8192 / 64 = 128.

KV bytes per token = 2 (K and V) x layers x KV heads x head dim x bytes per element = 2 x 80 x 8 x 128 x 2 = 327,680 bytes (320 KiB) per token at BF16, or 163,840 bytes with an FP8 KV cache.

A 32,768-token prefix (the context length in the shipped config) is therefore 32,768 x 327,680 = 10.74 GB. The derivation is the same as in our vLLM KV cache capacity planning article. For turning bytes into GPU counts, see the KV cache memory calculator.

Recompute cost. The model card lists 72.7B parameters, 70.0B of them non-embedding. Estimates used throughout (labelled assumptions):

• Linear-layer FLOPs per prefill token are about 2 x 70.0e9 = 140 GFLOP.
• Attention score and value FLOPs for the token at position p are about 4 x 64 heads x 128 x 80 layers x p = 2,621,440 x p. Summed over a causal prefix of S tokens, that is about 2,621,440 x S^2 / 2.
• Hardware is 8x H100 SXM. NVIDIA lists 1,979 TFLOPS BF16 with sparsity ^[4], so 989.5 TFLOPS dense per GPU. At an assumed 50% utilization, the node delivers 8 x 989.5e12 x 0.5 = 3.958e15 FLOP/s.

For S = 32,768: linear 32,768 x 140e9 = 4.59e15, plus attention 2,621,440 x 32,768^2 / 2 = 1.41e15, total 5.99e15 FLOP. 5.99e15 / 3.958e15 = 1.51 s of full-node prefill.

Hit-rate economics

Define hit rate as the fraction of prompt tokens served from cache. The table models a 16,384-token prompt on the node above. The uncached suffix still attends over the cached prefix, so compute saved is always slightly less than the hit rate.

The request rate ignores decode. The curve is convex: moving from 75% to 90% more than doubles prefill capacity.

The cost side is memory. Each 90%-hit session pins about 4.8 GB of KV until it is evicted. Rough capacity for the same node:

• HBM: 8 x 80 GB at 90% utilization, minus 145.4 GB of BF16 weights (72.7B x 2), leaves about 430 GB. That is roughly 1.31M tokens, or about 80 prefixes of 16k tokens. Activations and fragmentation are ignored, so this is optimistic.
• CPU DRAM: 2 TB of host memory (an assumed configuration) holds about 6.1M tokens, or about 372 such prefixes.
• NVMe: 4 x 30.72 TB drives (the top Micron 9550 capacity) hold about 375M tokens, or about 22,900 prefixes.

If more than about 80 sessions return within your typical reuse interval, an HBM-only cache evicts them before they come back and the hit rate collapses. Lower tiers exist to hold that working set.

When prefix caching does nothing

• No shared prefix. vLLM says APC brings no gain when "new queries do not share the same prefix with any of existing queries" ^[1].
• Decode-dominated work. APC also brings no gain when the engine "spends most of the time generating answers".
• Cache-busting templates. A timestamp, request ID, user name or shuffled tool list near the top of the prompt changes the first block's hash, and every later block misses. Put static content first and per-request content last.
• Truncation and rewriting. Agents that summarize or drop old turns miss on everything after the first edit.

Tiering beyond HBM: what each stack ships (October 2026)

Read on 2026-10-28; these projects ship monthly, so re-check before deploying.

First, Dynamo's configuration reference warns that if the CPU cache is smaller than the GPU KV cache, "KVBM churns (offloading blocks after every forward pass)". It suggests DYN_KVBM_CPU_CACHE_GB >= 100 and a disk cache at least as large as the CPU cache (Dynamo KVBM configuration, v1.6.1). Size the host tier as a multiple of HBM KV, not a fraction of it.

Second, vLLM's shorthand flags --kv-offloading-size (GiB) and --kv-offloading-backend have shipped since v0.11.1 (2025-11-18) and remain in v0.32.0 (2026-10-18), where native expands to the OffloadingConnector JSON above (vLLM blog, Jan 8, 2026; vllm/config/vllm.py at v0.32.0). The usage page still documents only the JSON form.

Bandwidth math: fetch vs recompute

Fetching wins when moving cached KV takes less time than rebuilding it. Same model and node, with these transfer assumptions:

• CPU DRAM to GPU: about 50 GB/s per GPU in one direction, the H100 figure the vLLM team measured for its connector ^[7], against NVIDIA's PCIe Gen5 rating of 128 GB/s ^[4]. With TP=8 each GPU holds one of the eight KV heads, so it loads one eighth of the bytes. We assume each GPU has its own x16 path to the host.
• NVMe: 14.0 GB/s sequential read per drive, a vendor spec ^[8], with four drives striped ideally.

Break-even bandwidth is KV bytes divided by recompute time: the slowest path that still ties. It falls as prefixes grow because attention FLOPs grow with S^2 while KV bytes grow with S. For this model, any path that sustains roughly 9 GB/s into the node, about 1.1 GB/s per GPU, wins on paper. Host DRAM wins by 44-56x. Even a single Gen5 NVMe wins, but NVMe data still has to cross PCIe into the GPU unless the two copies are pipelined.

That "on paper" carries weight. A September 2026 study by VU Amsterdam and IBM Research measured this tradeoff in vLLM on an H100 node ^[9]. Its findings:

• GPU-CPU copies sustained 54-56 GB/s, close to our assumption.
• For Llama 3.2 3B (114,688 bytes per token), the bandwidth needed to break even fell "from 23.2 GB/s at 1k tokens to 10.4 GB/s at 8k and 3.5 GB/s at 80k".
• An 8k prompt needed 77.8% prefix reuse to break even through the llm-d filesystem path. At 80k the threshold fell to 7.8%.
• "Below roughly 8k tokens ... no storage device we measured can load the prompt faster than the GPU rebuilds it."
• On Alibaba Bailian production-trace replays, the H100's average request fell below the SSD break-even point, so external caching gave no TTFT benefit there.

The authors conclude that external KV caching "should therefore be treated as a setup specific admission decision."

Two lessons for sizing:

1. Model shape decides. A 3B model on one GPU needs far more bandwidth per token of savings than a 72B model on eight GPUs, because each KV byte represents far less recomputation. Large GQA models, and MLA models with even smaller KV, are where storage tiers pay off soonest.
2. Fixed overheads dominate short prefixes. Lookups, file opens, staging and scheduling are invisible to bandwidth arithmetic. Calibrate the break-even length on your hardware and skip lower-tier loads below it.

The vLLM team's own single-request numbers fit this picture. For Llama-3.1-8B on H100, loading from CPU "reduces TTFT by X2-X22" depending on prompt size. With 10,000 requests of 512 tokens, throughput rose "by up to X9" with CPU hit rate ^[7]. These are project-run benchmarks, not independent results.

Throughput matters as much as latency: recomputed prefill takes GPU-seconds from other requests' decode, while a DMA copy mostly uses copy engines and the PCIe link. For pooled prefill, see our disaggregated prefill/decode analysis.

Grace Hopper and Grace Blackwell: where C2C changes the answer

On GH200, NVLink-C2C is rated at "up to 900 GB/s total bandwidth, 450 GB/s/dir". Each Grace CPU carries up to 512 GB of LPDDR5X at up to 546 GB/s, next to up to 96 GB of HBM3 at 3000 GB/s, and the GPU "can address up to 608 GB of memory within a superchip" ^[10]. These are vendor link ratings, not measured KV transfers, for the original HBM3 part; NVIDIA's current GH200 page lists HBM3e variants with up to 624 GB of combined fast memory, so check your SKU.

Per GPU link, the difference is large. Moving the 10.74 GB, 32k-token prefix above into one GPU takes about 215 ms at a measured 50 GB/s over PCIe, against 23.9 ms at the theoretical 450 GB/s per direction over C2C. The tier also changes character: 512 GB of LPDDR5X holds about 1.56M tokens at 320 KiB per token (3.1M with FP8 KV), about five times the HBM. Host memory becomes a working cache tier rather than a slow spill area.

On GB200, each superchip pairs one Grace CPU with two Blackwell GPUs. It has up to 480 GB of LPDDR5X at up to 512 GB/s, and 372 GB of HBM3E at 16 TB/s ^[11]. Its NVLink-C2C interface "delivers 900 GB/s of bidirectional bandwidth" ^[12]. Two GPUs share that CPU memory bandwidth, so plan with about half of it per GPU when both fetch at once. That is still several times a PCIe Gen5 link; the tighter constraint is capacity, roughly 240 GB of LPDDR5X per GPU versus 512 GB on GH200.

Operating it

Metrics to export and alert on:

• Hit rate by tier. vLLM exposes vllm:prefix_cache_queries and vllm:prefix_cache_hits as counters, plus vllm:kv_cache_usage_perc ^[13]. Track connector hits separately.
• TTFT split by hit and miss. Bucket by cached-prefix length to see whether lower-tier hits actually beat recompute.
• Churn. Compare blocks written to lower tiers against blocks later read back; write-heavy, read-light means wasted bandwidth.
• NVMe write volume. At 0% hit, the 72B node above prefills about 24,600 tokens/s (1.5 x 16,384). Writing all of it to disk is 24,600 x 327,680 B, about 8 GB/s or about 700 TB per day. Admission filters matter here. Dynamo's disk offload filter is on by default (DYN_KVBM_DISABLE_DISK_OFFLOAD_FILTER defaults to false) and its offload policies include frequency-based presence_lfu. vLLM's connector defaults offload_prompt_only to true ^[14]. Compare the result against your drives' rated endurance.
• Cold start. HBM and DRAM tiers are empty after a restart. Treat a surviving disk or remote tier as valid only if engine version, weights, dtype, block size and hash settings are unchanged.

Routing. A cache on one replica helps only requests routed to that replica. Dynamo's KV-aware router "Routes requests based on worker load and KV cache overlap". NVIDIA's README credits it with "2x faster TTFT", attributed to a Baseten benchmark on Qwen3-Coder 480B ^[15]. That is a vendor-reported result. For support and version-pinning questions, see our Dynamo support checklist. Without it, N replicas split each session across N caches; session affinity is the cheap fallback.

Security and tenancy

A shared prefix cache is a timing side channel: a faster first token tells tenant B that someone recently sent the same prefix. vLLM's answer is an optional per-request cache_salt that "is injected into the hash of the first block, ensuring that only requests with the same salt can reuse cached KV blocks". The docs say this "prevents timing-based attacks where an adversary could infer cached content by observing latency differences" ^[2].

Salt per tenant by default and share only deliberately public prefixes. Keep the cryptographic default hash on multi-tenant servers. Treat a shared L3 tier (Redis, S3, a shared filesystem) as derived customer data, with encryption, access control and retention to match.

Decision matrix

Whatever the stack: make prompts cache-friendly, measure real hit rate, size host memory above HBM KV, then add NVMe or remote tiers only where your calibrated break-even says loads beat recompute. GPU Smith helps teams run this kind of sizing on their own traffic and hardware.

Sources

1. Automatic Prefix Caching, vLLM documentation, page dated Oct 13, 2026 (accessed 2026-10-28)
2. Automatic Prefix Caching (design), vLLM documentation (accessed 2026-10-28)
3. KV Offloading usage, vLLM documentation, page dated Oct 26, 2026 (accessed 2026-10-28)
4. Inside vLLM's New KV Offloading Connector, vLLM Blog (Or Ozeri, Danny Harnik, IBM Research), Jan 8, 2026 (accessed 2026-10-28)
5. Metrics, vLLM documentation (accessed 2026-10-28)
6. SGLang: Efficient Execution of Structured Language Model Programs, Zheng et al., arXiv, v2 Jun 6, 2024 (accessed 2026-10-28)
7. SGLang HiCache Best Practices, SGLang documentation (accessed 2026-10-28)
8. LMCache documentation, The LMCache Team (accessed 2026-10-28)
9. ai-dynamo/dynamo README, NVIDIA, tag v1.6.1 (2026-10-27) (accessed 2026-10-28)
10. KVBM configuration reference, NVIDIA Dynamo documentation v1.6.1 (accessed 2026-10-28)
11. Qwen2.5-72B-Instruct config.json, Qwen team, Hugging Face (accessed 2026-10-28)
12. Qwen2.5-72B-Instruct model card, Qwen team, Hugging Face (accessed 2026-10-28)
13. NVIDIA H100 Tensor Core GPU, NVIDIA (accessed 2026-10-28)
14. NVIDIA Grace Hopper Superchip Architecture In-Depth, NVIDIA Technical Blog, Nov 10, 2022 (accessed 2026-10-28)
15. NVIDIA GB200 NVL72, NVIDIA (accessed 2026-10-28)
16. Micron 9550 NVMe SSD, Micron (accessed 2026-10-28)
17. Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs, Kanichai, De Matteis, Trivedi, arXiv, Sep 10, 2026 (accessed 2026-10-28)
18. NVIDIA GH200 Grace Hopper Superchip, NVIDIA (accessed 2026-10-28)
19. NVIDIA GB200 NVL72 Delivers Trillion-Parameter LLM Training and Real-Time Inference, NVIDIA Technical Blog, Mar 18, 2024 (accessed 2026-10-28)