Estimating TTFT From FLOPs: The Prefill Compute Floor

Why TTFT has a compute floor

Time to first token (TTFT) for a long prompt is dominated by prefill: the model has to run every prompt token through every layer before it can sample the first output token. Prefill processes those tokens in parallel, so it is limited by arithmetic, not by memory bandwidth. That gives a useful lower bound you can compute before you buy or benchmark anything:

TTFT floor ≈ (2 × matmul parameters × prompt tokens + attention FLOPs) ÷ (dense peak FLOPS × MFU)

Queueing, scheduling, network and tokenization sit on top of this number. A measured TTFT below the floor at a plausible MFU means a cache hit, a different precision or a different clock boundary; one far above it points to queueing or interference. For what TTFT measures and how to report it, see the TTFT, ITL and goodput definitions; this piece covers the arithmetic.

The Sarathi-Serve authors state the asymmetry: "Prefill iterations have high latency but saturate GPU compute due to parallel processing of the input prompt," while decode iterations "have low latency but also low compute utilization" ^[1].

You can check this with datasheet numbers. An H100 SXM lists 3.35 TB/s of memory bandwidth and 1,979 teraFLOPS of BF16 Tensor Core throughput with sparsity, so 989.5 dense ^[2]. The ridge point is 989.5 / 3.35 ≈ 295 FLOPs per byte. In prefill each BF16 weight (2 bytes) is read once and used for 2 FLOPs per prompt token, so arithmetic intensity roughly equals the number of tokens in the batch: above about 300 tokens, dense prefill is compute-bound on H100. FP8 doubles both peak and FLOPs per weight byte, so the crossover stays put.

The core formula

Linear layers: 2 FLOPs per parameter per token

Each weight does one multiply-accumulate (2 FLOPs) per token. Kaplan et al. write the forward-pass cost as C_forward ≈ 2N + 2·n_layer·n_ctx·d_model per token, "where the factor of two comes from the multiply-accumulate operation used in matrix multiplication" ^[3]. Here N counts non-embedding parameters.

For prefill, count only weights multiplied for every prompt token: exclude the embedding lookup, exclude the LM head (logits are needed only for the last position; under 0.1% here), and for mixture-of-experts use active parameters. Computed from the published Llama 3 hyperparameters (layers, model dimension, FFN dimension, 32/64 query heads, 8 key/value heads; Llama 3 Herd of Models, Table 3), the matmul parameter counts are:

Using the nominal 8B would overstate the linear term by about 15%.

Attention: the quadratic term, derived

The Q, K, V and O projections are inside N. Not in N are the two activation-by-activation matmuls, S = QKᵀ and O = S·V. For one layer, s prompt tokens and query width d_attn (query heads × head dimension):

• QKᵀ: s × s outputs, each a dot product of length d_attn: 2·s²·d_attn FLOPs.
• S·V: s × d_attn outputs, each a dot product of length s: 2·s²·d_attn FLOPs.

That is 4·s²·d_attn per layer without masking. FlashAttention-2 uses exactly this count, "4 · seqlen² · head dimension · number of heads," and divides it by 2 under a causal mask "to account for the fact that approximately only half of the entries are calculated" ^[4]. For a causal decoder prompt across L layers:

Attention FLOPs (prefill, causal) ≈ 2 · L · s² · d_attn

This equals Kaplan's per-token term 2·n_layer·n_ctx·d_attn summed over s tokens with n_ctx = s. Grouped-query attention shrinks the KV cache, not these FLOPs, which scale with query heads (Llama 3 70B: 64 × 128 = 8,192 = d_model). If your kernel computes the masked half anyway, use 4·L·s²·d_attn; sliding-window models need their own count.

When the attention term matters

The ratio of attention to linear FLOPs is (2·L·s²·d) / (2·N·s) = L·s·d / N. It grows with the prompt and is larger for smaller models:

At 8K the term is minor for 70B; at 32K it is not; by 128K it dominates. Kaplan et al. call the context term relatively small while d_model > n_ctx/12 (below about 49K tokens at d_model 4,096); the table shows that can still mean 30-60% at 32K, so keep it.

Datasheet trap: dense vs sparse TFLOPS

NVIDIA's headline Tensor Core numbers assume 2:4 structured sparsity, which dense LLM inference does not use; take half. On the H100 page the Tensor Core rows, including BF16, FP16 and FP8, carry an asterisk, and the footnote reads "* With sparsity" (NVIDIA H100, read 2026-10-28). The HGX page says it more plainly: "Specification in Sparse. Dense is ½ sparse spec shown" (NVIDIA HGX, read 2026-10-28).

*Sparse. HGX B200 and HGX B300 list the same FP16/BF16 (36 PFLOPS) and FP8/FP6 (72 PFLOPS) figures, footnoted sparse; the NVL72 row divides the GB200 NVL72 and GB300 NVL72 rack figures by 72 (pages read 2026-10-28). Same silicon, different power envelope: quote the system you buy.

A published check: Llama 3 405B pre-training reports 430 TFLOPs per GPU at 43% BF16 MFU on H100 ^[5]. 430 / 989.5 = 43.5%, so Meta used the dense figure as the denominator. Divide by the sparse 1,979 and you would get 21.7%.

Use the sparse number and every floor here halves: a team budgets 67 ms for a prefill that needs at least 133 ms.

FP8 dense throughput is twice BF16, so FP8 GEMMs halve the linear-term floor; whether attention also runs in FP8 depends on the kernel, so the examples assume BF16 attention. Quality and capacity trade-offs are in FP8 vs FP16 inference cost.

Choosing an MFU assumption

Model FLOPs utilization is "the ratio of the observed throughput to the theoretical maximum throughput if the benchmarked hardware setup were operating at peak FLOPS with no memory or communication overhead" ^[6]. Public anchors:

The 76% is in the paper body, not only the abstract: weight-gathered layouts are inefficient at low batch sizes, "but eventually they become the most efficient at high batch sizes, achieving 76% MFU when the communication overhead is almost negligible." That batch is about a million prompt tokens (512 × 2,048), not one user's TTFT; batch-1 prefill ran at 43%.

The formula reproduces Pope's batch-1 row (using the nominal 540B; the paper padded attention heads, adding 18B): 2 × 540B × 2,048 tokens = 2.21 × 10¹⁵ FLOPs. Spread over 64 chips × 275 TFLOPS bfloat16 at 43%, that is 0.29 s, matching Table 2 (a TPU check, not a GPU one).

Training MFU is not inference MFU: it includes backward passes and pipeline bubbles at a steady large batch, while single-request prefill loses time to launch gaps, non-matmul work, collectives and, for MoE, uneven expert loads.

Planning bracket used below: 40% (cautious) and 60% (optimistic) MFU for dense prefill of several thousand tokens. This is an assumption, not a measurement; expect less below about 2K tokens or with heavy sharding, and measure your stack.

Worked example 1: 8B dense, 8K prompt, one H100

Assumptions: Llama 3 8B, BF16, s = 8,192, one idle H100 SXM, no cache hit.

1. Linear FLOPs = 2 × 6.98 × 10⁹ × 8,192 = 1.144 × 10¹⁴
2. Attention FLOPs = 2 × 32 layers × 8,192² × 4,096 = 1.759 × 10¹³ (15.4% of linear)
3. Total = 1.319 × 10¹⁴ FLOPs
4. Dense peak = 989.5 × 10¹² FLOP/s

Memory check: reading 13.96 GB of weights at 3.35 TB/s takes about 4.2 ms, so compute binds. With FP8 GEMMs and BF16 attention the 100% ceiling becomes 57.8 + 17.8 = 76 ms; attention is now a quarter of the time, which is why FP8 helps long prompts less than 2×.

Worked example 2: 70B dense, 32K prompt, TP=8

Assumptions: Llama 3 70B, BF16, s = 32,768, eight H100 SXM at TP=8 (141 GB of weights do not fit one 80 GB card).

1. Linear FLOPs = 2 × 68.45 × 10⁹ × 32,768 = 4.486 × 10¹⁵
2. Attention FLOPs = 2 × 80 × 32,768² × 8,192 = 1.407 × 10¹⁵ (31% of linear)
3. Total = 5.893 × 10¹⁵ FLOPs
4. Aggregate dense peak = 8 × 989.5 = 7,916 TFLOP/s

An 8K prompt on the same node is 1.21 × 10¹⁵ FLOPs (0.38 s at 40%): four times the tokens cost 4.9 times the compute.

Tensor parallelism adds communication. A Megatron-style layout does two activation all-reduces per layer of s × d_model × 2 bytes ≈ 0.54 GB, so 85.9 GB over 80 layers; a ring all-reduce moves about 2(n−1)/n of that per GPU, ≈ 150 GB. NVIDIA's 900 GB/s NVLink figure counts both directions (18 links × 25 GB/s each way, per the Hopper architecture post), so at 450 GB/s per direction that is roughly 0.33 s if nothing overlaps compute. This rough illustration is why TP=8 prefill tends toward the low end of the bracket, and PCIe-only nodes lower still.

Worked example 3: MoE with active parameters

Assumptions: Mixtral 8x7B (32 layers, dim 4,096, FFN 14,336, 32 query heads, 8 KV heads, 8 experts, top-2; Mixtral, Table 1), s = 8,192, one H100 SXM, FP8 weights for GEMMs, BF16 attention.

Mixtral "has access to 47B parameters, but only uses 13B active parameters during inference." From the architecture, active matmul parameters are 32 × (41.9M attention + 2 × 176.2M experts) = 12.62B.

1. Linear FLOPs = 2 × 12.62 × 10⁹ × 8,192 = 2.067 × 10¹⁴, at FP8 dense 1,979 TFLOPS → 104.5 ms at 100%
2. Attention FLOPs = 1.759 × 10¹³ (same attention shape as the 8B example), at BF16 dense 989.5 TFLOPS → 17.8 ms at 100%
3. Floor = 122 ms at 100%, 204 ms at 60%, 306 ms at 40%

Two MoE caveats:

• Memory follows total parameters, compute follows active. 16,384 expert assignments (8,192 × top-2) over 8 experts means every expert is read: about 46.7 GB in FP8, 14 ms at 3.35 TB/s. Still under the compute time, but with far less margin than a dense model; short prompts can turn memory-bound.
• Routing costs utilization. The authors note overhead "due to the routing mechanism and due to the increased memory loads when running more than one expert per device." Plan at the low end of the bracket until measured.

Queueing, chunked prefill and batching effects

The floor is one request on an idle GPU. Production adds three things.

Queueing. Divide prompt FLOPs arriving per second by dense peak × MFU. As a rule of thumb (our assumption, not a sourced threshold), past about 0.6-0.7 expect P99 TTFT well above the floor.

Chunked prefill. vLLM's documentation says chunked prefill "allows vLLM to process large prefills in smaller chunks and batch them together with decode requests," that in V1 it "is enabled by default whenever possible," and that the scheduler "batches all pending decode requests before scheduling any prefill operations" (vLLM optimization and tuning, v0.32.0, released 2026-10-18). The token budget sets the trade-off: smaller max_num_batched_tokens values "(e.g., 2048) achieve better ITL," while "Higher values achieve better time to first token (TTFT)." With a 2,048-token budget a 32K prompt needs at least 16 iterations shared with decode work, all added to its TTFT.

Interference. DistServe frames prefill-decode interference as the reason to separate the phases and reports serving "7.4× more requests or 12.6× tighter SLO" against the systems it compared under its workloads ^[7]. That is a result on its own traces, not a general multiplier; see disaggregated prefill/decode break-even.

Prefix cache hits lower the floor

Only uncached prompt tokens need prefill. vLLM's automatic prefix caching "caches the KV cache of existing queries" so a new query can "skip the computation of the shared part," and the docs say it "only reduces the time of processing the queries (the prefilling phase)" ^[8].

New tokens still attend to cached ones: a token at position i costs about 4·L·d·i attention FLOPs. Example 2 with 28,672 cached tokens and 4,096 new:

1. Linear = 2 × 68.45 × 10⁹ × 4,096 = 5.61 × 10¹⁴
2. Attention = Σ 4 × 80 × 8,192 × i for i = 28,672 … 32,767 ≈ 3.30 × 10¹⁴
3. Total 8.91 × 10¹⁴, which is 15% of the uncached 5.89 × 10¹⁵. That is 0.23 s at 50% MFU, versus 1.49 s uncached.

The expected cache-hit length is often a more important input than the prompt length.

Validate with a benchmark

The formula is a hypothesis; a benchmark gives your stack's real MFU.

1. Isolate prefill: one request at a time, fixed prompt length, tiny output. vLLM's benchmark exposes --dataset-name random, --random-input-len, --random-output-len, --max-concurrency and --percentile-metrics (which can include ttft) ^[9].
2. Prevent cache hits: random prompts with --random-prefix-len 0, or serve with --no-enable-prefix-caching.
3. Sweep 2K, 8K and 32K prompts to expose the quadratic term.
4. Implied MFU = FLOPs(s) ÷ (median TTFT × dense peak), after subtracting a short-prompt baseline for network and tokenization.

Implied MFU above about 80% suggests a cache hit, a different precision or a smaller model. Around 20% (2× your 40% floor) usually means unoverlapped TP communication, a chunking token budget, weak attention kernels at that length, or queueing inside the clock. MFU falling with prompt length points at attention kernels. The same method turns the DGX Spark prefill speed figures in our review into implied MFU once precision and dense peak are fixed.

Checklist and spreadsheet formula

Inputs: L, d_attn, matmul parameters N (active, no embeddings), precisions, GPUs in the TP group, dense peak per precision, MFU, prompt length s and cached length c (0 if none).

new      = s - c
linear   = 2 * N * new
attn     = 2 * L * d_attn * (s^2 - c^2)      // causal; ≈ Σ 4·L·d·i for i = c..s
floor_s  = linear / (GPUs * dense_linear * MFU) + attn / (GPUs * dense_attn * MFU)

Red flags:

• Peak TFLOPS taken from an asterisked datasheet line.
• Total MoE parameters, or nominal size with embeddings.
• Training MFU applied to single-request TTFT.
• TTFT without prompt length, cache state, concurrency or clock boundary.
• No attention term above about 16K tokens.

GPU Smith runs this kind of prefill sizing as part of hardware selection work.

Sources

1. NVIDIA H100 Tensor Core GPU, NVIDIA, product page (accessed 2026-10-28)
2. NVIDIA HGX Platform, NVIDIA, product page (accessed 2026-10-28)
3. Scaling Laws for Neural Language Models, Kaplan et al., arXiv, January 2020 (accessed 2026-10-28)
4. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, Tri Dao, arXiv, July 2023 (accessed 2026-10-28)
5. The Llama 3 Herd of Models, Llama Team, Meta, arXiv, July 2024 (accessed 2026-10-28)
6. Efficiently Scaling Transformer Inference, Pope et al., arXiv, November 2022 (accessed 2026-10-28)
7. Mixtral of Experts, Jiang et al., arXiv, January 2024 (accessed 2026-10-28)
8. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, Agrawal et al., arXiv, March 2024 (accessed 2026-10-28)
9. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, Zhong et al., arXiv, January 2024 (accessed 2026-10-28)
10. Optimization and Tuning, vLLM documentation, v0.32.0 (released 2026-10-18; accessed 2026-10-28)
11. Automatic Prefix Caching, vLLM documentation, v0.32.0 (accessed 2026-10-28)
12. vllm bench serve, vLLM documentation, v0.32.0 (accessed 2026-10-28)
13. NVIDIA GB200 NVL72 and GB300 NVL72, NVIDIA, product pages (accessed 2026-10-28)
14. NVIDIA Hopper Architecture In-Depth, NVIDIA Technical Blog, 2022 (accessed 2026-10-28)