GPU SmithCOMPUTE ENGINEERING

roofline model llm inference decode

1 article

Decode Is Memory-Bound: Tokens per Second from HBM Bandwidth

Estimate an LLM decode tokens-per-second ceiling from GPU memory bandwidth: bytes per token, KV cache, batching, roofline ridge points, H100 to B300.

October 28, 2026• 17 min read
llm tokens per second memory bandwidth formulaestimate tokens per second from gpu bandwidthis llm inference memory bound or compute bound