MHA vs GQA vs MLA: How Attention Design Sets KV Cache Size

KV cache in one paragraph

During autoregressive decoding, every attention layer needs the keys and values of all earlier tokens in the sequence. Serving engines keep them in GPU memory instead of recomputing them: that is the KV cache. It grows linearly with tokens per sequence and again with the number of resident sequences. Shazeer's multi-query attention paper framed the cost precisely: incremental decoding is often slow "due to the memory-bandwidth cost of repeatedly loading the large 'keys' and 'values' tensors" ^[1]. The same tensors fill HBM as context or batch grows, and the attention design, not the parameter count, sets how many bytes each cached token costs.

The MHA baseline formula

In standard multi-head attention (MHA), each layer has n_heads query heads and the same number of key and value heads, each of width head_dim. Per cached token, per layer, you store one key vector and one value vector per KV head. The general per-token formula is:

KV bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_element

The factor 2 is keys plus values. DeepSeek-V2's comparison table gives the same expressions in elements: 2·n_h·d_h·l for MHA, 2·n_g·d_h·l for GQA with n_g groups, and 2·d_h·l for MQA ^[2]. Multiply elements by 2 bytes for BF16/FP16 or 1 byte for FP8, then by tokens per sequence and by sequences.

Worked example, MHA: Llama 2 7B. The config has 32 layers, 32 attention heads, 32 KV heads and hidden size 4096, so head_dim = 4096 / 32 = 128 (derived; the config has no head_dim field) ^[3].

• Elements per token: 2 x 32 x 32 x 128 = 262,144
• BF16: 524,288 bytes = 512 KiB per token
• FP8: 256 KiB per token

At the model's native 4,096-token context (max_position_embeddings: 4096 in the same config), one sequence holds 2 GiB of BF16 cache. At a hypothetical 128K (131,072-token) context, it would need 64 GiB per sequence.

MQA and GQA

Multi-query attention (MQA) keeps all query heads but shares a single key head and a single value head across them, "greatly reducing the size of these tensors and hence the memory bandwidth requirements of incremental decoding," with "only minor quality degradation" in Shazeer's experiments ^[1]. In KV-cache terms, kv_heads = 1. Pope et al. reported that, with appropriate partitioning, the lower memory requirement of multiquery attention "enables scaling up to 32x larger context lengths" on PaLM 540B ^[4].

Grouped-query attention (GQA) sits between the two. It uses "an intermediate (more than one, less than number of query heads) number of key-value heads," and the authors show that uptrained GQA "achieves quality close to multi-head attention with comparable speed to MQA" ^[5]. The experiments uptrained T5 checkpoints with 5% of original pre-training compute, so the quality claim applies to that setting. The paper also makes a sharding point that matters for multi-GPU serving: standard sharding "replicates the single key and value head by the number of model partitions," and GQA removes that waste ^[5].

For GQA, kv_heads in the formula is the config's num_key_value_heads, not num_attention_heads. Mixing the two up is the most common sizing error: it overstates Llama 3.1 cache by 4x (8B) or 8x (70B).

Worked example: Llama 3.1 8B

From the config: num_hidden_layers: 32, num_attention_heads: 32, num_key_value_heads: 8, head_dim: 128, max_position_embeddings: 131072, torch_dtype: bfloat16 ^[6]. Four query heads share each KV head. The meta-llama originals are gated; these counts match the Llama 3 paper's Table 3 ^[7].

• Elements per token: 2 x 32 x 8 x 128 = 65,536
• BF16: 131,072 bytes = 128 KiB per token
• FP8: 64 KiB per token
• One full 131,072-token sequence: 128 KiB x 131,072 = 16 GiB BF16, 8 GiB FP8

Same layers and head width as Llama 2 7B, a quarter of the KV heads, a quarter of the cache (derived).

Worked example: Llama 3.1 70B

From the config: num_hidden_layers: 80, num_attention_heads: 64, num_key_value_heads: 8, hidden_size: 8192, head_dim: 128 ^[8]. The mirror's config lists head_dim explicitly, and it matches 8192 / 64 = 128. Eight query heads share each KV head.

• Elements per token: 2 x 80 x 8 x 128 = 163,840
• BF16: 327,680 bytes = 320 KiB per token
• FP8: 160 KiB per token
• One 131,072-token sequence: 40 GiB BF16, 20 GiB FP8

For contrast (derived counterfactuals, not real checkpoints): with 64 KV heads (MHA) the same model would need 2,560 KiB per token and 320 GiB per 128K sequence; with 1 KV head (MQA), 40 KiB and 5 GiB.

MLA: caching a latent

Multi-head latent attention (MLA), introduced in DeepSeek-V2, does not reduce the number of heads. It compresses keys and values jointly into a low-rank latent vector and caches that latent instead of per-head K and V. MLA "guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector" ^[2]. Per-head keys and values are reconstructed from the latent, or the up-projections are absorbed into the query and output projections.

One complication is rotary position embeddings, which cannot be folded through the compression. DeepSeek therefore adds a small "decoupled" RoPE key shared across heads, which also has to be cached. The paper states the total: "DeepSeek-V2 requires a total KV cache containing (d_c + d_h^R)·l elements" per token, where d_c is the latent width and d_h^R the decoupled RoPE key width ^[2]. There is no factor 2: one latent serves both K and V. With d_c = 4·d_h and d_h^R = d_h/2, the paper notes its cache "is equal to GQA with only 2.25 groups" ^[2].

In config.json terms:

MLA KV bytes per token = layers x (kv_lora_rank + qk_rope_head_dim) x bytes_per_element

Worked example: DeepSeek-V3

From the config: num_hidden_layers: 61, num_attention_heads: 128, num_key_value_heads: 128, kv_lora_rank: 512, qk_rope_head_dim: 64, qk_nope_head_dim: 128, v_head_dim: 128, max_position_embeddings: 163840 ^[9]. Note that num_key_value_heads: 128 is misleading for an MLA model. Plugging it into the GQA formula gives the uncompressed cache, not the latent cache. The V3 technical report states that V3 adopts MLA, "thoroughly validated in DeepSeek-V2" ^[10].

• Elements per token: 61 x (512 + 64) = 35,136
• BF16: 70,272 bytes = 68.6 KiB per token
• FP8 (1 byte per element, scales ignored): 34.3 KiB per token
• One 131,072-token sequence: 8.58 GiB BF16, 4.29 GiB FP8

The MHA comparison uses the paper's formula 2·n_h·d_h·l with 128 heads and d_h = 128: 2 x 128 x 128 x 61 = 1,998,848 elements, or 3,904 KiB per token in BF16 and 488 GiB per 128K sequence. MLA stores about 1.8% of that (derived). Per layer, DeepSeek-V3 caches 576 elements per token; Llama 3.1 70B caches 2 x 8 x 128 = 2,048. So the 671B-parameter MoE stores less KV per token (68.6 KiB) than Llama 3.1 8B (128 KiB), even with nearly twice the layers.

Reconciling the 93.3% headline

The DeepSeek-V2 abstract says that, compared with DeepSeek 67B, V2 "reduces the KV cache by 93.3%" ^[2]. Our arithmetic from the configs:

• DeepSeek 67B (GQA): 95 layers, 8 KV heads, head_dim 8192 / 64 = 128, so 2 x 95 x 8 x 128 = 194,560 elements per token ^[11]
• DeepSeek-V2 (MLA): 60 layers x 576 = 34,560 elements per token ^[12]. This matches the 34.6K elements the paper's Table 9 lists for its large MLA model.

At equal precision that is an 82.2% reduction. The paper also says the deployed V2 used KV cache quantization "to further compress each element in its KV cache into 6 bits on average" ^[2]. With 6-bit V2 elements against a BF16 67B cache, the reduction becomes 1 - 25,920 / 389,120 = 93.3%. The same section says that "benefiting from MLA and these optimizations, actually deployed DeepSeek-V2 requires significantly less KV cache than DeepSeek 67B", which supports reading the headline as architecture plus quantization. The paper does not state the 67B cache precision, so the BF16 baseline and the split are our reconciliation. For capacity planning, use roughly 82% for the architectural share alone.

Concurrency and context consequences

The per-token number becomes a capacity number when you fix a KV budget. The table below assumes 100 GiB of HBM left for cache after weights and activations. That is an illustrative figure, not a property of any specific GPU or engine. Each sequence is a full 131,072 tokens.

All values derived from config.json fields using the formulas above; whole sequences rounded down.

Batch scaling is linear. Holding 32 concurrent 128K sequences needs 32 x 40 GiB = 1,280 GiB of BF16 cache for Llama 3.1 70B: about 640 GiB at FP8, versus 32 x 8.58 = 274.5 GiB for DeepSeek-V3 at BF16. A more typical chat mix of 64 sequences at 8,192 tokens needs 160 GiB (Llama 3.1 70B, BF16), 64 GiB (Llama 3.1 8B) or 34.3 GiB (DeepSeek-V3). The KV cache memory calculator covers sizing from prompt/output distributions rather than worst-case length.

Tensor parallelism changes the per-GPU picture

GQA heads shard across tensor-parallel ranks: at TP=8, each Llama 3.1 70B rank holds one KV head, or 40 KiB per token (derived). MLA's latent is a single shared vector, so plain TP does not split it. SGLang's team wrote that DeepSeek models "use MLA and only have one KV head," so tensor parallelism on 8 GPUs "will lead to duplicated KV cache" ^[13]. Their fix is data-parallel attention. SGLang's DeepSeek page describes it as reducing "KV-cache duplication across TP ranks," recommended for large-batch throughput and not for low-latency small batches ^[14]. Under plain TP=8, each DeepSeek-V3 rank carries the full 68.6 KiB per token, more than a TP=8 Llama 3.1 70B rank (derived).

Quantizing the KV cache

FP8 halves every BF16 figure above. vLLM exposes this as kv_cache_dtype, with fp8_e4m3 (CUDA 11.8+ and ROCm) and fp8_e5m2 (CUDA 11.8+). Scales default to 1.0 unless calibrated, and per-tensor scales apply separately to Q, K and V (vLLM quantized KV cache docs, page dated August 2026). That page gives no accuracy numbers. Validate FP8 KV on your own evaluation set, especially for long-context retrieval.

Engines do not always store "1 byte per element" exactly. vLLM's FlashMLA sparse backend (v0.11.0 docs; unchanged in the v0.32.0 source) defines an fp8_ds_mla format of 656 bytes per token per layer: 512 FP8 latent values, 16 bytes of float32 scales, and 64 RoPE values kept in BF16 because "this part is not quantized for accuracy" ^[15]. Across 61 layers that is 40,016 bytes (39.1 KiB) per token, about 14% more than the naive 34.3 KiB (derived). For the per-format accuracy and cost trade-offs, see FP16 vs FP8 vs INT4 inference costs.

Serving engines also differ in how they lay out MLA's latent. vLLM's MLA backend writes "the latent and rope to kv cache" as one 576-wide entry with a single KV head per layer. It uses an MHA-style compute path for prefill and an MQA-style, "data-movement friendly" path for decode ^[16]. A fallback that materializes per-head K and V would store far more.

Things the formula ignores

Sliding-window and hybrid attention. Some models apply full attention only on a subset of layers. Gemma 3 uses "5 local layers for every global layer," with local layers limited to a 1,024-token span ^[17]. For Gemma 3 27B, vLLM's design doc counts 52 sliding-window and 10 full-attention layers ^[18]. With 16 KV heads and head_dim 128 ^[19], the derived BF16 cache for one 128K sequence is 10 GiB for the global layers plus a 416 MiB cap for the local layers: about 10.4 GiB, versus 62 GiB if all 62 layers were global. gpt-oss-120b alternates "banded window and fully dense patterns" with a 128-token band and 8 KV heads of dimension 64 ^[20]. Its config lists 36 layers, half sliding_attention ^[21]. The 18 full layers cost 36 KiB per token in BF16, or 4.5 GiB at 128K (derived). These savings only materialize if the engine frees out-of-window blocks. vLLM does: it allocates per-token blocks and frees "blocks that are outside the sliding window." Its doc also notes that the layer-grouping scheme wastes some memory on padding for Gemma 3 ^[18].

Paging and prefix sharing. PagedAttention stores KV in fixed-size blocks with "near-zero waste in KV cache memory" and shares blocks "within and across requests" ^[22]. A partly filled last block adds a little per sequence; shared system prompts can push physical memory below the per-sequence sum.

Engine pools and reservations. Weights, activations and allocator headroom come first. Reconcile your arithmetic with the engine's startup report; the vLLM KV cache capacity planner walks through that check.

FAQ

Which config.json field gives the KV head count? num_key_value_heads. If it is absent, the model is MHA and KV heads equal num_attention_heads. If head_dim is absent, use hidden_size / num_attention_heads. For MLA models, ignore both and use kv_lora_rank + qk_rope_head_dim.

Is MLA always smaller than GQA? Per layer, MLA in DeepSeek-V3 stores 576 elements. A GQA layer stores 2 x kv_heads x head_dim, so GQA with 2 KV heads of width 128 (512 elements) would be smaller. The paper's own framing is "GQA with only 2.25 groups" ^[2]. Its claim is better quality than MHA at that size, not the smallest possible cache.

Does KV cache size depend on total parameters? No. A dense 70B model and a 671B MoE can be compared only through layers, KV heads (or latent width), head width and cache dtype. For DeepSeek-family deployment costs beyond the cache, see DeepSeek R1 self-hosting cost.

If you need these numbers turned into a GPU count for a specific workload, GPU Smith does that sizing work (contact).

Sources

• Fast Transformer Decoding: One Write-Head is All You Need — Noam Shazeer, arXiv, 2019-11-06, accessed 2026-10-28
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — Ainslie et al., arXiv, 2023-05-22, accessed 2026-10-28
• Efficiently Scaling Transformer Inference — Pope et al., arXiv, 2022-11-09, accessed 2026-10-28
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, arXiv, 2024-05-07, accessed 2026-10-28
• DeepSeek-V3 Technical Report — DeepSeek-AI, arXiv, 2024-12-27, accessed 2026-10-28
• Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., arXiv, 2023-09-12, accessed 2026-10-28
• Gemma 3 Technical Report — Gemma Team, Google, arXiv, 2025-03-25, accessed 2026-10-28
• gpt-oss-120b & gpt-oss-20b Model Card — OpenAI, arXiv, 2025-08-08, accessed 2026-10-28
• Llama 2 7B config.json (unsloth mirror) — Hugging Face, accessed 2026-10-28
• Llama 3.1 8B config.json (unsloth mirror) — Hugging Face, accessed 2026-10-28
• Llama 3.1 70B config.json (unsloth mirror) — Hugging Face, accessed 2026-10-28
• DeepSeek-V3 config.json — DeepSeek-AI, Hugging Face, accessed 2026-10-28
• DeepSeek-V2 config.json — DeepSeek-AI, Hugging Face, accessed 2026-10-28
• DeepSeek LLM 67B Base config.json — DeepSeek-AI, Hugging Face, accessed 2026-10-28
• Gemma 3 27B IT config.json (unsloth mirror) — Hugging Face, accessed 2026-10-28
• gpt-oss-120b config.json — OpenAI, Hugging Face, accessed 2026-10-28
• Hybrid KV Cache Manager (design doc) — vLLM, dated 2026-08-19, accessed 2026-10-28
• Quantized KV Cache — vLLM, dated 2026-08-03, accessed 2026-10-28
• MLA common backend (v0.9.2 API docs) — vLLM, accessed 2026-10-28
• FlashMLA sparse backend (v0.11.0 API docs) — vLLM, accessed 2026-10-28
• SGLang v0.4: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs — LMSYS, 2024-12-04, accessed 2026-10-28
• DeepSeek V3/V3.1/R1 Usage — SGLang, accessed 2026-10-28
• The Llama 3 Herd of Models — Llama Team, Meta, arXiv, 2024-07-31, Table 3, accessed 2026-10-28