Long-Context LLM Serving Hardware: 128K to 1M Tokens

What changes between 8K, 128K and 1M tokens

Moving from 8K to 128K or 1M tokens changes three costs at different rates:

1. KV cache memory grows linearly with tokens. Every resident token stores a key and a value vector in every attention layer.
2. Prefill compute grows quadratically in the attention layers. Meta explained Llama 3's late long-context training this way: "the compute in self-attention layers grows quadratically in the sequence length" ^[1].
3. Decode becomes a KV-read problem. Each generated token reads its sequence's entire cache. At long context that read outweighs the weights, and batching does not amortize it.

Concurrency collapses from hundreds of chat sessions per node to dozens at 128K and a handful at 1M, each costing more per token. The questions become HBM bytes per GPU, HBM bandwidth, and NVLink domain size, mapped below onto vLLM's tensor parallel (TP), decode context parallel (DCP), prefill context parallel (PCP) and KV offload. For sizing against your own traffic distribution, use the KV cache memory calculator.

KV cache arithmetic at long context

For standard attention, cache bytes per token are:

KV bytes/token = 2 (K and V) x layers x KV heads x head_dim x bytes per element

The number that matters is KV heads, not query heads. Grouped-query attention (GQA) uses "an intermediate (more than one, less than number of query heads) number of key-value heads" and the GQA authors report that uptrained GQA reaches "quality close to multi-head attention with comparable speed to MQA" ^[2]. Llama 3 8B, 70B and 405B all use 8 KV heads, with 32, 80 and 126 layers respectively, and model dimensions of 4,096, 8,192 and 16,384 ^[1]. Their head dimension is 128 (8,192 / 64 query heads for 70B; 16,384 / 128 for 405B).

Applying the formula with BF16 cache (2 bytes per element). This is our arithmetic from the published hyperparameters:

Llama 3 supports "a context window of up to 128K tokens" ^[3], so its 1M column is arithmetic only. The Qwen row uses the published config (28 layers, 4 KV heads, hidden size 3,584 over 28 query heads) (config.json); its card claims "Full 1,010,000 tokens and generation 8192 tokens" ^[4].

GQA buys a constant factor, not a different curve: the 64-to-8 head reduction cuts the 70B cache 8x at every length, but one 128K request still uses as much cache as sixteen 8K requests. Latent attention compresses further; DeepSeek-V2's Multi-head Latent Attention compresses the KV cache "into a latent vector" and reports a 93.3% KV reduction relative to DeepSeek 67B ^[5]. Architecture choice is the cheapest long-context hardware decision you will make.

Sanity check: the Qwen 7B cache at 1,010,000 tokens is 57,344 x 1,010,000 = 57.9 GB; add 7.61B BF16 parameters (about 15.2 GB) and prefill buffers, and the card's "At least 120GB VRAM (total across GPUs)" for 1M tokens is plausible ^[4].

Because vLLM's PagedAttention reaches "near-zero waste in KV cache memory" ^[6], engine capacity tracks this formula once weights and runtime buffers are subtracted; the vLLM KV capacity planner covers reading the engine's reported pool.

Prefill compute and TTFT at 128K and beyond

Kaplan et al. give forward cost per token as C_forward = 2N + 2 x n_layer x n_ctx x d_attn (N = non-embedding parameters; the second term is attention against the context), noting that for "contexts and models with d_model > n_ctx/12, the context-dependent computational cost per token is a relatively small fraction of the total compute" ^[7]. For d_model = 8,192 that boundary is n_ctx = 98,304; long-context serving lives beyond it.

For a causal prefill of T tokens, token t attends to t predecessors, so summing the per-token cost gives approximately:

Prefill FLOPs ~= 2 x N x T  +  n_layer x d_attn x T^2

GQA does not shrink the quadratic term, because d_attn is the query width: GQA saves memory, not attention FLOPs. The two terms cross at T = 2N / (n_layer x d_attn).

Worked example, a Llama 3 70B-class model (N ~ 70e9, 80 layers, d_attn 8,192, so n_layer x d_attn = 655,360). This is our arithmetic:

Crossover: 2 x 70e9 / 655,360 ~ 213,600 tokens. Beyond roughly 200K tokens, attention is most of the prefill work for this model shape.

Converting to a TTFT floor: the H200 SXM is listed at 1,979 TFLOPS BF16 and 3,958 TFLOPS FP8, both "With sparsity" ^[8], so dense peaks are half of that: 989.5 and 1,979 TFLOPS. Across eight GPUs at 100% of dense BF16 peak:

• 128K prompt: 2.96e16 / (8 x 989.5e12) = 3.7 s
• 1M prompt: 8.67e17 / (8 x 989.5e12) = 110 s

These are floors; achieved utilization is below peak and TTFT scales up in proportion. A 1M-token prompt on one 8-GPU H200 node takes minutes unless the architecture avoids full attention. The Qwen 1M model card reports that its sparse-attention vLLM fork "achieves a 3 to 7 times speedup for sequences up to 1M tokens" ^[4]. That is a vendor claim, and as of 2026-10-28 the card still requires building QwenLM's dev/dual-chunk-attn vLLM branch. Upstream vLLM merged a dual-chunk flash-attention backend in May 2025 (PR #11844) for its since-removed V0 engine; the v0.32.0 backend registry has no dual-chunk backend, only the dual-chunk RoPE, so the card's sparse-attention path still needs that branch.

Chunked prefill does not reduce these FLOPs. It slices them so that decodes of other requests keep running in between. vLLM chunks prefills "based on the remaining max_num_batched_tokens" ^[9], and the Qwen card recommends --max-num-batched-tokens 131072 with --enable-chunked-prefill for 1M serving. For how TTFT and inter-token latency (ITL) interact under load, see TTFT and ITL metrics.

Memory-bound decode at long context

A decode step for a batch reads the model weights once, then reads each sequence's entire KV cache. At long context the second read dominates and grows with batch size. Worked example on 8x H200 (aggregate 8 x 4.8 TB/s = 38.4 TB/s ^[8]), 70B weights in FP8 (~70 GB) and BF16 KV at 327,680 bytes per token. This is our arithmetic, a bandwidth floor that ignores compute and kernel efficiency:

The last two rows read the same bytes per step because both fill the same KV budget, yet the 128K workload produces 16x fewer tokens per second from the same hardware. The bill scales with resident tokens, not requests. When HBM is full of KV, step time approaches HBM capacity divided by HBM bandwidth:

• H100 SXM: 80 GB / 3.35 TB/s ~ 23.9 ms ^[10]
• H200 SXM: 141 GB / 4.8 TB/s ~ 29.4 ms
• B200 (DGX B200): 1,440 GB / 64 TB/s per 8 GPUs ~ 22.5 ms ^[11]
• GB200 superchip: 372 GB / 16 TB/s ~ 23.3 ms ^[12]
• MI355X: 288 GB / 8 TB/s ~ 36 ms ^[13]

Capacity sets how many long sessions fit; bandwidth sets the ITL they get. More capacity without proportionally more bandwidth makes a full GPU slower per token.

Tensor parallel and KV duplication

TP shards the KV cache along the head dimension H. Because H is fixed by the architecture, vLLM warns that "when we continue to increase the tensor parallel size, the KV cache for each GPU will be duplicated for tp_size / H times" ^[14].

The last three rows are vLLM's own case studies. Duplication silently divides effective KV capacity: an 8-GPU MLA deployment at TP 8 holds one-eighth of the context it could.

Decode context parallel (DCP)

DCP shards the KV cache along the token dimension instead. In vLLM you add -dcp <size>; "size does not increase the number of GPUs we need to launch, but just reduces the KV cache duplication", it "should lie in the range of [1, tp_size/H]", and larger values cut duplication at higher communication cost ^[14]. The flag is --decode-context-parallel-size (default 1) ^[9].

• Sharding is interleaved along T; the knob is now --cp-kv-cache-interleave-size, replacing --dcp-kv-cache-interleave-size ^[9].
• Block accounting changes. "a physical block of 16 tokens with DCP=4 represents 64 tokens", reported as effective_attention_block_size ^[14]. Capacity dashboards that multiply physical blocks by block size will under-report by the DCP factor.
• Keep DCP inside the NVLink domain where possible. For Kimi-K2 at TP 16, vLLM suggests -dcp 8 as a compromise that leaves 2x duplication but keeps "DCP communication only ... inside one node".
• Order: "try to increase -tp size until you get satisfactory performance, and then add -dcp." DCP supports "both MLA and GQA models."

Worked example: an MLA model at TP 8 with a 300 GB per-node KV budget holds 300 / 8 = 37.5 GB of unique KV, because every GPU stores a full copy of its share. With -dcp 8 the same node holds 300 GB of unique KV, 8x more resident context with no new hardware.

Prefill context parallel and ring attention

DCP does nothing for the quadratic prefill floor; for that the sequence itself must be split across GPUs. vLLM describes two strategies ^[14]:

1. Partial query, full key/value. Each GPU computes Q/K/V for its chunk, K/V are gathered, and each GPU computes attention for its own queries. This works when the full K/V of one request still fits per GPU.
2. Partial query, partial key/value. When the request is "too long", K/V stay sharded and are passed chunk by chunk using ring attention.

Ring Attention overlaps "the communication of key-value blocks with the computation of blockwise attention", supports sequences "up to device count times longer" than prior memory-efficient Transformers, and reports "millions of tokens context size" ^[15]. Llama 3's training stack instead all-gathers K and V, accepting exposed latency for attention-mask flexibility ^[1].

Status in vLLM v0.32.0 (released 2026-10-18): the deployment guide still says of both prefill strategies, "Both approaches are under active development", while the engine arguments include --prefill-context-parallel-size / -pcp, which "expands the process world size but does not increase the KV-cache shard count" ^[9]. Treat PCP as something to qualify on your pinned version, not as a default production path.

Prefill CP moves K/V for the whole sequence in every layer. On NVLink (900 GB/s per H200; 1.8 TB/s GPU-to-GPU on HGX B200/B300 ^[16]) that can be overlapped; across scale-out NICs it is much harder to hide, which makes long-context prefill a strong argument for a larger NVLink domain.

Offload tiers

vLLM exposes --kv-offloading-size ("Size of the KV cache offloading buffer in GiB"), with native or lmcache backends ^[9]. What matters is the link the KV crosses:

Our arithmetic (bytes / listed bandwidth); real one-direction PCIe throughput is lower.

Offload is for parking, not streaming. Reading a 1M-token 70B-class context (343.6 GB) from LPDDR5X at 512 GB/s takes 0.67 s per pass, unusable per decode step. It pays when idle sessions, shared prefixes or agent histories are parked and pulled back once, because reloading beats recomputing: a 128K prefill for the 70B example is at least 2.96e16 FLOPs, 15 s on one H200 at dense FP8 peak, versus about 48 ms over C2C. GH200 offers "up to 624GB of combined CPU and GPU fast memory" ^[17]; a GB200 NVL72 rack lists "17 TB LPDDR5X | 14 TB/s" ^[12].

Hardware choices

Vendor-listed figures as of 2026-10-28. NVIDIA's H200 page carries the note "Preliminary specifications. May be subject to change."

Figures are from the vendor pages linked above (HGX for B300 capacity, NVIDIA's Blackwell Ultra post for its bandwidth); the "fit" column is our judgment.

Ask which constraint binds first. Capacity binds when one request approaches a GPU group's KV budget: a 1M-token 70B-class BF16 cache (343.6 GB) cannot share one GB200 superchip's 372 GB with weights, so it needs DCP or FP8 KV. Bandwidth binds when many long sessions are resident. Domain size binds when TP x DCP x PCP needs more than 8 GPUs per request, which is the case for rack-scale NVLink.

Cost and concurrency model

Concurrency per 8x H200 node for the 70B example, with our assumptions: 1,128 GB HBM total, 70 GB FP8 weights, and 10% of HBM (112.8 GB) reserved for activations, CUDA graphs and runtime, leaving 945 GB for KV. TP 8 equals the 8 KV heads, so there is no duplication.

FP8 KV halves bytes per token. vLLM's --kv-cache-dtype accepts fp8 among other options ^[9]. Validate accuracy at your target length before relying on it.

At full occupancy step time is roughly constant, so cost per output token scales almost linearly with context: a 128K session costs roughly 16x an 8K session per generated token, before prefill. Meter long-context tiers by resident token-seconds, not requests. Prefill adds at least 3.7 s of a full node per uncached 128K prompt; if prompts are mostly unique (document Q&A) rather than repeated prefixes (agent loops), prefill capacity may set the node count. When the two phases need different hardware, separating them becomes worth costing; see disaggregated prefill/decode break-even.

Decision tree

1. Check the architecture first: KV heads and attention type (GQA, MLA, sliding or sparse). Compressed KV or sparse attention changes every number above.
2. Compute KV per request at p95 context against HBM after weights and reserve. If it fits one GPU with headroom, use plain replicas; otherwise raise TP up to the number of KV heads.
3. If TP exceeds KV heads (always for MLA models, often for 4-8 KV-head GQA models), add -dcp up to tp/H. Keep DCP within one NVLink domain.
4. If TTFT at p95 prompt length misses the SLO after chunked prefill tuning, evaluate prefill context parallel on your pinned vLLM version, or disaggregate prefill onto its own pool.
5. If sessions are long-lived or share prefixes, add a CPU KV tier (--kv-offloading-size); prefer Grace-class C2C when reload latency matters.
6. If one request needs more than 8 GPUs to hold or prefill, budget for a rack-scale NVLink domain rather than stretching CP across InfiniBand.
7. Price by resident tokens, with separate SLOs and quotas for the long-context tier.

GPU Smith can help run this arithmetic against a specific model and traffic profile.

Sources

1. vLLM, "Context Parallel Deployment" (v0.32.0 docs, released 2026-10-18, accessed 2026-10-28): https://docs.vllm.ai/en/v0.32.0/serving/context_parallel_deployment/
2. vLLM, "Engine Arguments" (v0.32.0 docs, accessed 2026-10-28): https://docs.vllm.ai/en/v0.32.0/configuration/engine_args/
3. Liu, Zaharia, Abbeel, "Ring Attention with Blockwise Transformers for Near-Infinite Context", arXiv 2310.01889: https://arxiv.org/abs/2310.01889
4. Ainslie et al., "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints", arXiv 2305.13245: https://arxiv.org/abs/2305.13245
5. Llama Team, Meta, "The Llama 3 Herd of Models", arXiv 2407.21783 (abstract): https://arxiv.org/abs/2407.21783
6. Llama Team, Meta, "The Llama 3 Herd of Models", HTML v3 (Table 3, §3.3.2, §3.4.2): https://arxiv.org/html/2407.21783v3
7. Kaplan et al., "Scaling Laws for Neural Language Models", arXiv 2001.08361 (Table 1): https://arxiv.org/abs/2001.08361
8. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", arXiv 2309.06180: https://arxiv.org/abs/2309.06180
9. DeepSeek-AI, "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model", arXiv 2405.04434: https://arxiv.org/abs/2405.04434
10. Qwen, "Qwen2.5-7B-Instruct-1M" model card and config.json (accessed 2026-10-28): https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-1M
11. NVIDIA, "H200 Tensor Core GPU" (accessed 2026-10-28): https://www.nvidia.com/en-us/data-center/h200/
12. NVIDIA, "H100 Tensor Core GPU" (accessed 2026-10-28): https://www.nvidia.com/en-us/data-center/h100/
13. NVIDIA, "DGX B200" (accessed 2026-10-28): https://www.nvidia.com/en-us/data-center/dgx-b200/
14. NVIDIA, "HGX Platform" (accessed 2026-10-28): https://www.nvidia.com/en-us/data-center/hgx/
15. NVIDIA, "GB200 NVL72" and "Grace Hopper Superchip" (accessed 2026-10-28): https://www.nvidia.com/en-us/data-center/gb200-nvl72/ and https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/
16. AMD, "Instinct MI355X" (accessed 2026-10-28): https://www.amd.com/en/products/accelerators/instinct/mi350/mi355x.html
17. NVIDIA Technical Blog, "Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era" (2025-08-22, updated 2025-09-24; accessed 2026-10-28): https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era