Decode Is Memory-Bound: Tokens per Second from HBM Bandwidth

The one-line model

During autoregressive decode, each generated token requires a forward pass that reads the active model weights and the request's KV cache out of GPU memory. If memory bandwidth is the bottleneck, the speed limit for a single stream is:

tokens/s per stream ≤ memory bandwidth ÷ bytes read per token

and its inverse is the floor on inter-token latency (ITL):

ITL ≥ bytes read per token ÷ memory bandwidth

For a 70B-parameter dense model with FP8 weights (about 70 GB) on an H100 SXM at 3.35 TB/s ^[1], the weights alone give 3.35 TB/s ÷ 70 GB ≈ 48 tokens/s, which is an ITL floor of about 21 ms. No kernel tuning gets past that number at batch size 1. Only reading fewer bytes per token or adding bandwidth does.

The rest of this article covers what goes into "bytes read per token", how batching changes the picture, where the roofline says decode turns compute-bound, and how far real systems fall short of the ceiling. All throughput figures below are derived upper bounds computed from vendor peak specifications. They are not measurements.

Why decode is memory-bound

The roofline model compares a kernel's arithmetic intensity (FLOPs per byte moved from memory) with the hardware's ridge point (peak FLOP/s ÷ peak bytes/s). Below the ridge, a kernel is limited by bandwidth. Above it, the limit is compute. The LLM Inference Unveiled survey builds its whole analysis of inference techniques on this framework, starting from the question of "why LLMs are memory-bound".

Arithmetic intensity of a decode GEMM

Take one weight matrix of d_in × d_out stored at b bytes per parameter, multiplied by a batch of B token vectors. The work is 2·B·d_in·d_out FLOPs. The weight traffic is b·d_in·d_out bytes, and at small B the activations are negligible next to it. So:

intensity ≈ 2B ÷ b FLOP/byte

With FP8 weights (b = 1), intensity is 2B. At batch 1 that is 2 FLOP/byte.

Ridge points from datasheets

NVIDIA lists Tensor Core figures "with sparsity". The Blackwell pages say outright that "Dense is ½ sparse spec shown" ^[2]. The table below applies the same halving to Hopper.

Inputs: H100, H200, HGX, DGX B200, GB200 NVL72, NVIDIA's Blackwell Ultra architecture post (Table 2, "Dense FP8: 5 PFLOPS" for both Blackwell and Blackwell Ultra) and the AMD MI355X page. Ridge points and crossover batches are our arithmetic. The HGX and DGX B200/B300 pages list 72 PFLOPS FP8 per 8 GPUs, footnoted sparse, so 4.5 PFLOPS dense per GPU; the GB200 and GB300 NVL72 pages list 720 PFLOPS per 72 GPUs, 5 PFLOPS dense per GPU, which is the architecture post's figure (all read 2026-10-28). The gap reflects configuration and power envelope, not an error; use the row for the system you buy.

At batch 1, an H100 running an FP8 decode GEMM sits at 2 FLOP/byte against a ridge of 591. The GPU uses roughly 0.3% of its dense FP8 compute before the memory system saturates. The How To Scale Your Model inference chapter reaches the same conclusion for TPUs: matmuls become compute-bound only once the per-replica token batch exceeds a critical size set by the FLOPs-to-bandwidth ratio (240 tokens for bf16 on TPU v5e). The analysis in Pope et al., which reports 29 ms per token at low batch on PaLM 540B with int8 weights, rests on the same weight-loading argument.

Attention never crosses the ridge

Weights are shared across the batch. KV cache is not, because each sequence reads only its own keys and values. With grouped-query attention, each KV element is used by g query heads, so attention intensity is about 2g ÷ b_kv FLOP/byte no matter how large the batch gets. Llama 3 70B has 64 query heads and 8 KV heads ^[3], which gives g = 8. With an FP8 KV cache that is about 16 FLOP/byte, far below every ridge in the table. This is the main reason long-context decode stays memory-bound even at large batch.

Bytes per token: weights

Weight bytes per token = active parameters × bytes per parameter.

• Dense models read every parameter for every token. A 70B model needs ~140 GB at BF16, ~70 GB at FP8 and ~39 GB at NVFP4.
• NVFP4 is not a flat 0.5 bytes. NVIDIA describes it as "one 4-bit value plus minor overhead of one FP8 scale per 16 values (4.5 bits per value)" ^[4]. That works out to 0.5625 bytes per parameter, or 39.4 GB for 70B.
• Mixture-of-experts (MoE) models read only the routed experts for each token. DeepSeek-V3 has "671B total parameters with 37B activated for each token" ^[5]. At batch 1 that means ~37 GB of FP8 reads per token. The full 671 GB still has to sit in memory.

Bytes per token: KV cache

KV bytes read per token = context length × KV bytes per token of context. For standard multi-head or grouped-query attention:

KV bytes per context token = 2 (K and V) × layers × KV heads × head dim × bytes per element

For Llama 3 70B (80 layers, 8 KV heads, model dimension 8,192 over 64 heads = head dim 128, per Table 3) with an FP8 cache:

2 × 80 × 8 × 128 × 1 = 163,840 bytes (160 KiB) per context token

DeepSeek-V3 uses multi-head latent attention. Its config lists kv_lora_rank 512, qk_rope_head_dim 64 and 61 layers ^[6], so the cached latent is (512 + 64) × 61 × 2 bytes = 70,272 bytes (~69 KiB) per context token at BF16. That is our derivation, assuming the latent is cached in BF16. For full capacity math across architectures, use the KV cache memory calculator.

At batch 1, KV barely matters for short chats. It dominates once you batch many long contexts.

Batching changes the math

One decode step produces one token for each of the B sequences in the batch. Weights are read once per step, and each sequence's KV is read once per step:

bytes per step = active weights + Σ (context_i × KV bytes per token) aggregate tokens/s ≤ B × bandwidth ÷ bytes per step per-stream tokens/s ≤ bandwidth ÷ bytes per step

Batching amortizes the weight read and leaves the KV read untouched. Aggregate throughput rises, per-stream speed falls, and HBM capacity eventually caps how large B can grow. That capacity cap is covered in our LLM inference hardware sizing guide.

Datasheet bandwidth table

As of October 2026, peak bandwidth per GPU from vendor pages:

Notes on the Blackwell rows:

• The DGX B200 page lists "1,440 GB total, 64 TB/s HBM3e bandwidth". The HGX and DGX B300 pages have no memory bandwidth row.
• The explicit per-GPU figure comes from NVIDIA's Blackwell Ultra post: "Bandwidth: 8 TB/s per GPU, 2.4x improvement over H100 (3.35 TB/s)". Its Table 2 lists 8 TB/s for both Blackwell and Blackwell Ultra. B300 added capacity, not bandwidth.
• AMD's comparison chart on the MI355X page lists the "B200 SXM5 180GB" at 7.7 TB/s. That is a competitor's figure; this article uses NVIDIA's 8 TB/s (DGX B200: 64 TB/s ÷ 8).

Worked example A: 70B dense at FP8

Inputs: 70 GB of FP8 weights, FP8 KV at 160 KiB per context token, and an 8,192-token context.

Batch 1, single GPU

Bytes per token = 70 + 1.34 = 71.34 GB.

At 128K context, KV adds 21.5 GB and the H200 ceiling drops from 67.3 to 52.5 tokens/s. On an 80 GB H100, 70 GB of weights leaves about 10 GB for KV and runtime buffers, so even modest batches at 8K context do not fit.

Batching on one GPU (8K context per sequence)

Going from batch 1 to batch 32 on H200 raises the aggregate ceiling 20× and lowers per-stream speed by 37%. Batch 32 is still memory-bound: 2 × 70e9 × 32 = 4.5e15 FLOPs per step takes 2.3 ms at 1,979 TFLOPS dense FP8, against 23.5 ms of memory time. The capacity limit arrives long before the ridge.

Tensor parallelism across two H100s

With TP=2, each GPU reads half the weights and half the KV heads: 35.67 GB per step, giving a 93.9 tokens/s ceiling (10.6 ms). Megatron-style TP adds two all-reduces per layer, or 160 per token for 80 layers. Assumption: small-message all-reduce latency of 5-25 µs adds 0.8-4.0 ms per token, which lowers the ceiling to roughly 68-87 tokens/s. Measure this on your own interconnect. The point is that TP divides the bandwidth term and adds a latency term that does not shrink.

Worked example B: large MoE (DeepSeek-V3 on 8× H200)

DeepSeek-V3 has 58 MoE layers (61 layers, of which the first 3 are dense), 256 routed experts plus 1 shared expert per MoE layer, 8 experts per token, hidden size 7,168 and expert FFN size 2,048 (config.json). Derived from those values:

• Parameters per expert = 3 × 7,168 × 2,048 ≈ 44.0M
• Routed expert parameters = 44.0M × 256 × 58 ≈ 654B, which is nearly all of the 671B
• Everything else (attention, dense layers, shared experts, embeddings) ≈ 17B

Batch 1: ~37.5 GB of FP8 reads per token. Spread perfectly across eight H200s (38.4 TB/s aggregate), that is a ~1,000 tokens/s ceiling. This number is not useful on its own. At batch 1, 61 layers of expert dispatch, combine and collective operations dominate the step, and real single-stream speed is a small fraction of it.

Batching: different tokens route to different experts, so the weight read grows with batch until every expert is touched. Assumption: routing is uniform and independent. Then the expected number of distinct experts per layer is 256 × (1 − (1 − 8/256)^B). KV uses MLA at ~69 KiB per context token with 8K context.

MoE batching amortizes weights only after the batch is large enough to cover most experts. Up to that point, bytes per step grow almost as fast as batch size. It also helps to look at bytes per generated token: 37.5 GB at batch 1 versus 9.1 GB at batch 64, compared with 70 GB for the dense 70B model at batch 1. The 671 GB residency is what decides how many GPUs you buy. The active parameter count, combined with batch size, decides speed.

Realistic efficiency factors

The ceiling assumes the full peak bandwidth goes to useful bytes. Real systems lose some of it in several places:

• Achievable HBM bandwidth. Peak figures are theoretical. Streaming kernels reach less, and decode kernels add launch gaps and tails between layers.
• End-to-end bandwidth utilization. Databricks defines model bandwidth utilization (MBU) as achieved bandwidth ÷ peak, with achieved = (model parameter size + KV cache size) ÷ time per output token. It reported "a higher MBU of 60% on 2xH100-80GBs" at batch size 1, compared with 55% on 4×A100-40GB, and noted that "MBU decreases as batch size increases" ^[12]. That measurement is from 2023 software. Assumption for planning: budget for 50-80% of peak at small batch on a well-tuned modern stack, and confirm with your own runs.
• Communication. TP all-reduces and MoE all-to-all add latency that does not shrink with bandwidth (see example A).
• Speculative decoding works as a multiplier on the ceiling. One weight read can verify several drafted tokens, so tokens per weight read rise with the acceptance rate, at the cost of the draft model's own reads. The draft model's cost is its own sizing question.

Applying a 0.6-0.8 factor to example A's H200 batch-1 ceiling gives a planning range of about 40-54 tokens/s. That is an estimate, not a benchmark.

Checking the ceiling against measurements

Turn your measured ITL into achieved bandwidth and compare it with peak:

achieved bandwidth = bytes per step ÷ measured ITL

Illustrative: if a 70B FP8 model on one H200 at batch 1 with 8K context shows a 20 ms ITL, achieved bandwidth = 71.3 GB ÷ 0.020 s = 3.57 TB/s, which is 74% of 4.8 TB/s. That is healthy. At 35% you should look for a misconfiguration, such as weights not running in the precision you think, CUDA graphs disabled, a CPU-bound scheduler, or unintended TP communication. Measure ITL and TPS the way our ITL and TPS metrics article describes. Use steady-state decode only, excluding time to first token.

When the model fails

• Prefill. Prompt processing multiplies weights by thousands of tokens at once, so it runs far above the ridge and is compute-bound. Bandwidth math says nothing about time to first token.
• Very large batches. Once B·2/b passes the ridge (about 200-310 tokens of FP8 batch in the table above), weight GEMMs become compute-bound. Attention still is not.
• Small models. An 8B BF16 model (16 GB) has a 500 tokens/s ceiling on an 8 TB/s GPU, which is 2 ms per token. Fixed per-step costs (kernel launches, sampling, scheduling, detokenization) become a large share of that 2 ms, so measured numbers fall well short of the ceiling.
• CPU-bound serving. Schedulers, structured-output masks and tokenization can stall the GPU between steps.
• Multi-GPU and MoE. Communication latency, expert load imbalance and pipeline bubbles are outside the model entirely.

FAQ

Why does DGX Spark feel slow on large models? Its 128 GB of LPDDR5x has 273 GB/s of bandwidth ^[11]. A 70B model at FP8 gives a ceiling of 273 ÷ 70 ≈ 3.9 tokens/s. NVFP4 (39.4 GB) gives about 6.9 tokens/s, and an 8B BF16 model about 17 tokens/s. Large capacity lets big models fit. It does not make them fast. Our DGX Spark review covers the rest of the trade-offs.

LPDDR vs HBM: how big is the gap? DGX Spark's 256-bit LPDDR5x gives 273 GB/s. H100 SXM HBM3 is about 12× that, and the 8 TB/s parts about 29×. Batch-1 decode speed follows the same ratio.

What is B300 memory bandwidth? NVIDIA states 8 TB/s per Blackwell Ultra GPU, the same as B200, with 288 GB of HBM3e ^[13]. The B300 therefore holds more KV cache than a B200, but its batch-1 decode ceiling for the same model is the same.

Does FP4 double decode speed over FP8? In the bandwidth model, weight bytes drop by about 1.8× (4.5 versus 8 bits per value), so a weight-dominated decode ceiling rises by up to ~1.8×. KV bytes only shrink if the KV cache is quantized as well.

If you want a second pair of eyes on a sizing model like this before a hardware purchase, GPU Smith does that work.

Sources

• NVIDIA H100 Tensor Core GPU — NVIDIA, accessed 2026-10-28
• NVIDIA H200 Tensor Core GPU — NVIDIA, accessed 2026-10-28
• NVIDIA DGX B200 — NVIDIA, accessed 2026-10-28
• NVIDIA HGX Platform — NVIDIA, accessed 2026-10-28
• NVIDIA DGX B300 — NVIDIA, accessed 2026-10-28
• NVIDIA GB200 NVL72 — NVIDIA, accessed 2026-10-28
• NVIDIA GB300 NVL72 — NVIDIA, accessed 2026-10-28
• Inside NVIDIA Blackwell Ultra: The Chip Powering the AI Factory Era — NVIDIA Technical Blog, 2025-08-22 (updated 2025-09-24), accessed 2026-10-28
• AMD Instinct MI355X GPU — AMD, accessed 2026-10-28
• NVIDIA RTX PRO 6000 Blackwell Server Edition — NVIDIA, accessed 2026-10-28
• NVIDIA DGX Spark — NVIDIA, accessed 2026-10-28
• Introducing NVFP4 for Efficient and Accurate Low-Precision Inference — NVIDIA Technical Blog, 2025-06-24, accessed 2026-10-28
• LLM Inference Unveiled: Survey and Roofline Model Insights — Yuan et al., arXiv, 2024, accessed 2026-10-28
• Efficiently Scaling Transformer Inference — Pope et al., arXiv, 2022-11-09, accessed 2026-10-28
• All About Transformer Inference (How To Scale Your Model) — Austin et al., accessed 2026-10-28
• LLM Inference Performance Engineering: Best Practices — Databricks, 2023-10-12, accessed 2026-10-28
• The Llama 3 Herd of Models — Meta, arXiv, 2024, accessed 2026-10-28
• DeepSeek-V3 Technical Report — DeepSeek-AI, arXiv, 2024-12-27, accessed 2026-10-28
• DeepSeek-V3 config.json — DeepSeek-AI on Hugging Face, accessed 2026-10-28