vllm
9 articles
GPU Memory Snapshot Recovery for Inference: Compatibility
A 2026 qualification guide to GPU inference snapshots: driver and chip compatibility, attributed cold-start measurements, a runnable validation protocol, and a worksheet comparing restore with warm replicas.
FP16 vs FP8 vs INT4 Inference Cost: Quality and GPU Fit
A 2026 guide to quality-adjusted GPU inference cost, comparing FP16/BF16, FP8 and exact INT4/FP4 recipes with a compatibility crosswalk, integer GPU-fit examples, acceptance gates and a Python calculator.
KV Cache Memory Calculator: Context, Concurrency and GPUs
A 2026 guide to architecture-aware KV cache sizing: calculate bytes per token, replay prompt/output distributions, reconcile vLLM and TensorRT-LLM capacity, and validate replica count against latency targets.
Batch vs Real-Time LLM Inference: GPU Fleet Sizing Guide
A 2026 guide to batch vs real-time LLM inference: size private GPU fleets to deadlines, benchmark SLO-qualified throughput, and compare spare-capacity reuse, reserved pools and nightly windows.
NVFP4 vs MXFP4 vs FP8 Inference Compatibility Guide
A 2026 compatibility guide to NVFP4, MXFP4, and FP8 across vLLM, SGLang, TensorRT-LLM, NVIDIA and AMD GPUs, with native versus emulated paths, artifact checks, and a reproducible validation protocol.
LLM Inference Metrics: TTFT, ITL, TPS and Goodput
A 2026 worksheet for comparing LLM inference benchmarks: define TTFT, ITL, per-user and aggregate TPS, normalize workload and timing boundaries, and reject unsupported goodput claims.
Triton 26.08 Upgrade Runbook: CUDA 13.4, Canary and Rollback Testing
A 2026 upgrade runbook for Triton Inference Server 26.08 covering CUDA 13.4.1 compatibility, container image selection, backend validation, canary testing, and rollback criteria.
Disaggregated Prefill/Decode Private Fleet Break-Even
A 2026 break-even model for private disaggregated prefill/decode fleets, covering SLO goodput, KV transfer, idle pool-hours, HBM opportunity cost, and conditional routing.
NVIDIA Dynamo 1.4.2 Enterprise Support Checklist
A 2026 enterprise readiness guide to Dynamo 1.4.2 artifacts, backend and CUDA pins, the NIXL loader fix, failure coverage, pilot metrics, and contract questions.