GPU SmithCOMPUTE ENGINEERING

dynamo planner autoscaling ttft itl

1 article

Autoscaling LLM Inference: Queue, KV Cache, Concurrency

Choose LLM autoscaling signals: vLLM queue depth, KV cache usage and in-flight concurrency, with HPA ratio math, KEDA triggers, Dynamo Planner and warmup.

October 28, 2026• 14 min read
autoscale vllm kubernetes kedahpa custom metric vllm num_requests_waitingllm autoscaling queue depth kv cache usage