Chunked Prefill Tuning: Token Budget, TTFT and ITL

Why prefill and decode interfere

Each iteration of a continuous-batching LLM server runs one forward pass over whatever the scheduler admitted. Prefill processes a whole prompt at once and is compute-bound. Decode produces one token per running sequence per step, reading weights and the growing KV cache for very little compute, so it is bandwidth-bound. In the Sarathi-Serve authors' terms, prefill has high latency and saturates GPU compute, while decode has low latency per step and poor compute utilization ^[1].

Problems start when both kinds of work share a GPU. Suppose a 30,000-token RAG prompt arrives while 100 chat sessions are streaming. If the scheduler runs the whole prompt in one iteration, every one of those 100 streams waits for it. The paper calls this a generation stall: "one or more prefills are scheduled in between consecutive decode iterations of a request" ^[2]. Users see it as an inter-token latency (ITL) spike whose tail can reach seconds while the median looks fine. DistServe reaches the same diagnosis: colocating the phases causes "strong prefill-decoding interferences" and forces a system to favour either time to first token (TTFT) or time per output token (TPOT) when SLOs are tight ^[3].

If you need precise definitions of TTFT, ITL, TPOT and goodput before tuning, start with TTFT, ITL and goodput definitions.

What chunked prefill does

Chunked prefill caps tokens per iteration: the scheduler admits running decodes (one token each), then fills the rest of a fixed budget with prompt tokens, splitting long prompts across iterations. Sarathi-Serve calls these "chunked-prefills" combined with "stall-free scheduling". New requests join a running batch without pausing the decodes, and the near-uniform iteration size also reduces pipeline bubbles under pipeline parallelism ^[1].

The paper's headline results are 2.6x higher serving capacity for Mistral-7B on one A100 and up to 3.7x for Yi-34B on two A100s, both against vLLM. It also reports up to 5.6x end-to-end serving capacity for Falcon-180B with pipeline parallelism, all under tail-latency constraints (arXiv 2403.02310; published at USENIX OSDI '24). These compare against 2024 vLLM, which now ships chunked prefill itself; they do not describe today's gap.

Chunking costs something. Each chunk attends over the KV cache of all earlier chunks, so splitting a prompt raises HBM reads "even though the computational cost is unchanged". The authors measured at most about 25% prefill overhead at a 512-token chunk on Yi-34B and almost none at a 2048-token budget. Chunk alignment also matters: a 257-token chunk can take 32% longer than a 256-token chunk because of tile quantization ^[2]. Use multiples of 256, and go small only if the ITL SLO requires it. The paper itself used 512 for strict SLOs and 2048 for relaxed ones ^[2].

The knob: max_num_batched_tokens in vLLM

In vLLM V1, "chunked prefill is enabled by default whenever possible". With it enabled, "the scheduling policy prioritizes decode requests": the scheduler batches all pending decodes first. A prefill that does not fit into max_num_batched_tokens is chunked automatically ^[4]. That makes --max-num-batched-tokens the per-iteration token budget B. It covers decode tokens and prefill tokens together.

The documentation states the trade-off directly:

• Smaller values, such as 2048, "achieve better ITL because there are fewer prefills slowing down decodes".
• Higher values "achieve better time to first token (TTFT) as you can process more prefill tokens in a batch".
• For throughput, vLLM recommends max_num_batched_tokens > 8192, "especially for smaller models on large GPUs".
• Setting it equal to max_model_len is close to the old V0 policy, except that decodes are still prioritized ^[4].

Defaults depend on hardware. The docs page gives no single default because there isn't one. In the v0.32.0 source, EngineArgs.get_batch_defaults picks the budget from the device's total memory and from whether you launched the OpenAI server (vllm serve) or the offline LLM class ^[5]. The same function is unchanged at v0.29.0:

A code comment explains the A100 exclusion: a large budget reduced A100 throughput ^[5]. The consequence is that the same vllm serve command starts H100 and A100 servers at different points on the TTFT/ITL curve, and a config copied across GPU generations without an explicit --max-num-batched-tokens silently changes B.

Related scheduler fields in v0.32.0 ^[6]:

• --long-prefill-token-threshold (default 0, meaning off) marks a prompt as "long" above the threshold so its per-step share can be capped. The cap is skipped when the request is alone in the batch.
• --max-num-queued-tokens rejects new requests with HTTP 503 once the prompt tokens waiting in prefill exceed a limit. The docstring suggests sizing it as target_TTFT × prefill_throughput.
• scheduler_reserve_full_isl (default True) checks that the full input length fits in the KV cache before admitting a request, "rather than only checking the first chunk". This prevents over-admission and KV thrashing under chunked prefill.
• max_num_batched_tokens must be at least max_num_seqs. Otherwise config validation raises an error.

SGLang equivalents

SGLang splits the same concept across several flags ^[7]:

The documented default of None is resolved at startup: in v0.5.22, handle_gpu_memory_settings sets it from GPU memory: 2048 below 20 GB (comment: T4, 4080); 4096 below 35 GB (A10, 4090, 5090) and below 60 GB (A100 40GB, L40); 8192 below 90 GB (H100, A100) and below 160 GB (H20, H200); 16384 above that (B200, MI300) ^[8]. The same function then reserves activation memory in proportion to the chunk size before it computes mem_fraction_static. Raising --chunked-prefill-size by hand therefore also shrinks the KV pool. For OOM during long prefills they recommend a smaller chunk size ^[7].

The most important difference from vLLM is --enable-mixed-chunk. Its help text reads "Enabling mixing prefill and decode in a batch when using chunked prefill", and it defaults to False ^[9]. With it off, the v0.5.22 scheduler runs a pending prefill batch first ("Run prefill first if possible") and decodes only on steps with no new prefill, so every chunk step pauses all running streams for a full prefill step ^[10]. Equal chunk sizes therefore do not imply equal ITL cost across engines. When porting a tuned budget from one engine to the other, re-run the sweep rather than copying the number.

Reading the effect: a worked example

A first-order model predicts the curves before you spend GPU hours. All inputs below are hypothetical; replace them with your own measurements.

Assume one replica with 64 sequences decoding. A decode-only step takes 15 ms, and each extra prefill token adds 0.025 ms to the step (a marginal prefill rate of 40,000 tokens/s). Decodes use 64 of the B tokens, so each step carries B - 64 prefill tokens.

Step time when the budget is full: t = 15 ms + (B - 64) × 0.025 ms

Service-time TTFT for one 8,000-token prompt with an empty queue:

• B = 1024: 8 full chunks of 960 (7,680 tokens) plus a 320-token tail. That is 8 × 39.0 + (15 + 8.0) = 335 ms.
• B = 2048: 4 × 1,984 = 7,936 tokens plus a 64-token tail. That is 4 × 64.6 + (15 + 1.6) = 275 ms.
• B = 4096: 4,032 + 3,968 tokens = 115.8 + (15 + 99.2) = 230 ms.
• B = 8192: one step = 15 + 8,000 × 0.025 = 215 ms.

On an idle server the TTFT gap is modest (335 vs 215 ms); the ITL cost to everyone else is not (39 vs 218 ms per affected step).

Under load, capacity decides TTFT. Now add 4 requests/s with 6,000-token prompts, an offered prefill load of 4 × 6,000 = 24,000 tok/s. Prefill utilization is offered load divided by capacity:

Large budgets raise prefill capacity by spreading the fixed 15 ms decode cost over more prompt tokens, which is why vLLM recommends B > 8192 for throughput. Under an ITL p99 SLO of 80 ms, only B = 2048 satisfies both constraints here. Under 50 ms, only B = 1024 meets ITL, and it runs prefill at 97.5% utilization. No setting meets both, and that is the signal that chunking alone is not enough (see below).

The model omits growth of decode time with batch and context, the rising cost of later chunks (each attends over all earlier KV), chunking overhead at small budgets, and bursty arrivals. Use it to choose the sweep range, never the final value.

A tuning protocol

1. Freeze everything except B. Fix the model and quantization, engine version, tensor/pipeline parallel sizes, max_model_len, gpu_memory_utilization, prefix-caching setting and max_num_seqs.

2. Build a representative trace. Use production prompt/output length distributions, not one fixed length. vllm bench serve supports --request-rate with Poisson arrivals and a --burstiness factor, where values below 1 make arrivals burstier ^[11].

3. Sweep B over multiples of 256: 1024, 2048, 4096, 8192, 16384. Run each value at three or more request rates spanning 50-95% of the knee you expect from the arithmetic above.

4. Report percentiles and goodput together. For example:

   vllm serve $MODEL --max-num-batched-tokens 2048 --max-num-seqs 256
   vllm bench serve --model $MODEL --dataset-name random \
     --random-input-len 6000 --random-output-len 300 --num-prompts 2000 \
     --request-rate 4 --burstiness 1.0 \
     --percentile-metrics ttft,tpot,itl --metric-percentiles 50,90,99 \
     --goodput ttft:2000 tpot:50 --save-result
   

   --goodput accepts only request-level keys (ttft, tpot, e2el) in milliseconds, following DistServe's definition ^[11]. ITL percentiles therefore come from --percentile-metrics, while TPOT stands in for ITL in the goodput count. The default is p99 only.

5. Scrape server metrics during each run ^[12]. Counters are exposed with a _total suffix.

   • vllm:iteration_tokens_total: histogram of tokens per engine step. If most steps sit at B, prefill is saturated.
   • vllm:num_preemptions and vllm:kv_cache_usage_perc: KV pressure.
   • vllm:num_requests_waiting: queue depth.
   • vllm:request_queue_time_seconds and vllm:request_prefill_time_seconds: split TTFT into waiting and computing.

6. Pick by constraint, then by goodput. Discard any B whose ITL p99 misses the SLO at the target rate. Among the rest, take the highest goodput. Within noise, prefer the smaller B for burst headroom.

Workload shapes

Prefix caching changes the arithmetic. vLLM's automatic prefix caching lets a new query "skip the computation of the shared part", and it "only reduces the time of processing the queries (the prefilling phase)" ^[13]. With high hit rates, offered prefill load is only the uncached tokens, so measure the hit rate (vllm:prefix_cache_hits / vllm:prefix_cache_queries) before you size B from raw prompt lengths.

Interaction with batch size and KV capacity

B, max_num_seqs and KV capacity form one system:

• Decodes consume budget. With 256 running sequences and B = 1024, only 768 tokens per step are left for prefill. Raising max_num_seqs without raising B quietly cuts prefill capacity and raises TTFT.
• KV pressure causes preemption. When the KV cache fills, vLLM preempts requests. In V1 the default mode is RECOMPUTE, and "preemption and recomputation can adversely affect end-to-end latency". The documented remedies are to raise gpu_memory_utilization, add tensor parallelism, or lower max_num_seqs or max_num_batched_tokens ^[4]. A recomputed request re-enters prefill, so preemption storms show up as TTFT and ITL regressions.
• Larger B costs activation memory. SGLang makes this explicit by reserving memory in proportion to chunk size before it sizes the KV pool ^[8]. In vLLM, compare the KV capacity reported in the startup log across budgets rather than assuming it is constant.

PagedAttention's block-level KV management is what makes this fine-grained admission practical. The original paper reported near-zero KV waste and 2-4x throughput over FasterTransformer and Orca in 2023 ^[14]. To size the KV pool for a target concurrency, use the vLLM KV cache capacity planner.

When chunking is not enough

Chunked prefill moves work along one dial; it cannot create capacity. When the ITL SLO forces a B whose prefill capacity is close to offered load, no budget satisfies both SLOs. In a sweep, the signs are:

• every B that meets ITL p99 runs iteration_tokens near the budget and shows a rising num_requests_waiting;
• TTFT p99 is dominated by queue time rather than prefill time;
• the TTFT and ITL Pareto frontier across your sweep sits entirely outside the SLO box.

At that point you have three options: add colocated replicas, accept a looser SLO, or separate the phases. vLLM's disaggregated prefilling is documented as "experimental and subject to change". It lets you tune TTFT and ITL separately and stops vLLM inserting "prefill jobs during the decoding of one request". It also states that disaggregation "DOES NOT improve throughput" ^[15]. The economics, including the KV transfer cost and the second minimum pool, are covered in when to disaggregate prefill and decode.

Rollout and regression gates

• Canary one replica with the new B and compare TTFT p99, ITL p99 and preemption rate against control replicas over at least one daily peak. Roll back if either percentile regresses beyond a tolerance set before the test.
• Alert on the mechanism, not just the SLO. A sustained shift of the iteration_tokens histogram toward B, or a rising preemption rate, leads SLO breaches.
• Pin B explicitly in every manifest. Both engines derive defaults from GPU memory, so a node-pool change can alter B without a config diff.
• Re-run the sweep on engine upgrades. Scheduler fields change between releases (--long-prefill-token-threshold-adaptive and --max-num-active-seqs exist in v0.32.0 but not v0.29.0).

Checklist

GPU Smith can run this sweep against your own trace and hardware as part of an inference sizing engagement.

Sources

1. vLLM, "Optimization and Tuning", docs (latest), accessed 2026-10-28. https://docs.vllm.ai/en/latest/configuration/optimization/
2. vLLM, vllm/engine/arg_utils.py, tag v0.32.0 (get_batch_defaults). https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/engine/arg_utils.py
3. vLLM, vllm/config/scheduler.py, tag v0.32.0. https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/config/scheduler.py
4. vLLM, vllm/v1/metrics/loggers.py, tag v0.32.0. https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/v1/metrics/loggers.py
5. vLLM, vllm/benchmarks/serve.py, tag v0.32.0. https://github.com/vllm-project/vllm/blob/v0.32.0/vllm/benchmarks/serve.py
6. vLLM v0.32.0 release (2026-10-18). https://github.com/vllm-project/vllm/releases/tag/v0.32.0
7. vLLM, "Automatic Prefix Caching", docs (latest). https://docs.vllm.ai/en/latest/features/automatic_prefix_caching/
8. vLLM, "Disaggregated Prefilling (experimental)", docs (latest). https://docs.vllm.ai/en/latest/features/disagg_prefill/
9. Agrawal et al., "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve", arXiv 2403.02310 (v3, 2024-06-17). https://arxiv.org/abs/2403.02310 (full text: https://arxiv.org/html/2403.02310v3)
10. USENIX OSDI '24 presentation page, Sarathi-Serve. https://www.usenix.org/conference/osdi24/presentation/agrawal
11. SGLang, "Server Arguments", docs. https://docs.sglang.io/advanced_features/server_arguments.html
12. SGLang, python/sglang/srt/arg_groups/memory_hook.py, tag v0.5.22. https://github.com/sgl-project/sglang/blob/v0.5.22/python/sglang/srt/arg_groups/memory_hook.py
13. SGLang, python/sglang/srt/arg_groups/fields/schedule.py, tag v0.5.22. https://github.com/sgl-project/sglang/blob/v0.5.22/python/sglang/srt/arg_groups/fields/schedule.py
14. SGLang v0.5.22 release (2026-10-19). https://github.com/sgl-project/sglang/releases/tag/v0.5.22 Scheduler loop: python/sglang/srt/managers/scheduler.py @ v0.5.22, https://github.com/sgl-project/sglang/blob/v0.5.22/python/sglang/srt/managers/scheduler.py
15. Zhong et al., "DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving", arXiv 2401.09670. https://arxiv.org/abs/2401.09670
16. Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention", arXiv 2309.06180 (SOSP 2023). https://arxiv.org/abs/2309.06180