GPU SmithCOMPUTE ENGINEERING

tensorrt-llm fp8 kv cache

1 article

FP8 KV Cache Quantization: Capacity Gains and Accuracy Risk

FP8 KV cache halves bytes per cached token. Exact vLLM, SGLang and TensorRT-LLM flags, scale handling, a worked Qwen3-32B example, and an eval gate.

October 28, 2026• 16 min read
fp8 kv cache vllmkv cache quantization accuracy long context--kv-cache-dtype fp8_e4m3