LLM Inference Roofline Model: FLOPS vs HBM Worksheet

Executive Summary

An LLM inference roofline model screens whether a graphics processing unit (GPU) upgrade has enough arithmetic capacity and high-bandwidth memory (HBM) bandwidth to merit a workload-specific pilot. It places an upper bound on attainable arithmetic throughput, or equivalently a lower bound on execution time. It does not predict delivered tokens per second. The original framework counts operations against traffic reaching external memory after cache effects; an HBM worksheet must preserve that boundary. ^{[1] [2]}

The calculation needs work F, HBM traffic Q, compatible peak compute C, and peak bandwidth BW. Arithmetic intensity is F/Q in floating point operations per byte; the device ridge point is C/BW in the same units. The optimistic device time is max(F/C, Q/BW). Below the ridge, additional compute alone does not improve this bound. Above it, additional bandwidth alone does not improve it. These statements describe the simplified envelope, with the workload and byte count held fixed. ^{[1] [3]} Low-batch decode and prompt prefill therefore require separate worksheets: decode processes a single token per active request per iteration, while prefill processes the prompt in parallel. ^[4]

Two hypothetical examples, calculated below rather than measured, illustrate the difference. A fictional device with 800 TFLOP/s compatible dense compute and 3 TB/s HBM bandwidth has a 266.67 FLOP/byte ridge. A decode step doing 16 GFLOP and moving 16.5 GB has a 5.5 ms device lower bound. A prefill doing 262.144 TFLOP and moving 20 GB has a 327.68 ms lower bound. These are arithmetic consequences of explicitly assumed inputs, not hardware performance claims. Doubling the limiting resource halves the corresponding bound in these examples; real speedup still requires measurement. ^[1]

Official specification labels matter as much as the arithmetic. AMD MI350X lists 2.3 PFLOPS BF16 matrix performance separately from 4.6 PFLOPS with structured sparsity; dense work must use a compatible ceiling. ^[5] Kernel selection, key-value (KV) cache traffic, quantization metadata, host dispatch and inter-GPU communication can change the applicable model. ^{[6] [7] [8]} The procurement outcome should be benchmark this candidate, reject the stated claim, or need more data. Require workload-matched traces, an appropriate profiler roofline, and unprofiled throughput at the chosen service-level objective (SLO). MLCommons distinguishes aggregate output throughput from per-user speed, reinforcing why a device bound cannot establish an interactive service result. ^[9]

Introduction and Background

Peak floating point operations per second (FLOPS) answer a narrow question: how much compatible arithmetic an accelerator can theoretically execute. A buyer needs a different answer: whether that resource constrains the specified inference workload. Roofline analysis connects the questions through arithmetic intensity, the amount of work performed for each byte moved at a defined memory boundary. Its performance envelope joins a bandwidth-dependent region to a compute ceiling. ^[2]

This report uses that connection as a procurement screen, not a product leaderboard or a new measured benchmark. GPU Smith's perspective here is that of an adjacent, independent engineering advisor. The reader's deliverable is a reproducible worksheet and a bounded decision about a pilot, rather than a recommendation based on advertised FLOPS alone.

Separate the workload into prompt processing and token generation before comparing devices. Sarathi-Serve describes prefill as parallel prompt processing and decode as successive single-token-per-request iterations. Its scheduling analysis also shows why mixing those phases matters to serving behavior. ^[4] A proposal that averages them into one tokens-per-second claim can obscure which phase the buyer actually needs to improve.

The analysis is anchored to specifications and documentation accessed on October 31, 2026; retrieval dates are recorded in the input ledger. These are source snapshots, not a claim that every document was issued on that date. No client results, commercial conversion rates or measured GPU speedups are asserted.

The scope is deliberately bounded to a device-level HBM roofline. Modal's explanation explicitly excludes system latency from the diagram. ^[10] A research implementation of this style of optimistic bound, Quail, likewise acknowledges that implementations cannot satisfy all its assumptions. ^[11] Accordingly, a promising lower bound opens an evidence request; it cannot close a purchasing decision.

Key Changes in How to Read an Upgrade Proposal

Replace a headline FLOPS comparison with a compatible ceiling

The numerator must describe the instructions the implementation actually uses. BF16, or bfloat16, FP16, or 16-bit floating point, and lower-precision formats require distinct entries. NVIDIA's profiling guidance specifically requires selecting the roofline section that matches the instruction data path. A tensor-core ceiling should not be assigned indiscriminately to every kernel in a model. ^[3]

Precision and sparsity are separate choices. AMD's MI350X page lists BF16 matrix throughput without structured sparsity separately from its structured-sparsity throughput. ^[5] The worksheet should require evidence that the deployed representation and kernels qualify for the selected peak. A weight file advertised by bitwidth is insufficient evidence about the execution path.

The source ledger should preserve the exact wording around each compute figure:

• Device identity: SKU, form factor and number of accelerators represented.
• Compute mode: input datatype, matrix or vector path, and accumulation mode where documented.
• Sparsity qualification: dense value or the precise condition attached to a sparse value.
• Memory scope: HBM bandwidth per device rather than a node aggregate.
• Operating configuration: power policy, clocks and software versions used in the pilot.
• Evidence state: official specification, analytical assumption or measurement.

These are proposed purchasing controls. They turn an ambiguous comparison into an auditable input set. NERSC notes that practical execution conditions, including power constraints and programming models, affect the relevance of nominal peaks. ^[2]

Separate memory capacity from memory bandwidth

Capacity determines whether the required representation and working state fit; bandwidth constrains how quickly bytes can move. Google's TPU v5p documentation distinguishes nominal physical HBM from user-allocatable memory reported by tools. This is a useful warning about interpreting capacity labels even when the purchase under consideration is a GPU. ^[12] A capacity increase may enable a different batching or placement policy, but that improvement must be modeled as a workload change.

HBM traffic is also different from allocated bytes. A retained weight or cache block can be allocated once but read repeatedly, while cache reuse can prevent some reads from reaching HBM. Triton's matrix multiplication tutorial explicitly changes block order to promote reuse. ^[13] Therefore, use an allocation worksheet for fit and a traffic worksheet for roofline analysis; reconcile them without treating them as interchangeable.

Replace one inference number with phase-specific objectives

Use time to first token (TTFT) for initial responsiveness and an explicitly defined subsequent-token latency measure for generation. MLCommons describes system throughput as total output tokens across concurrent users and separately considers per-user speed. ^[9] Specify the buyer's percentile, request distribution and arrival process before interpreting either metric. These are pilot requirements chosen by the buyer, not parameters supplied by a GPU datasheet.

The Buyer’s Roofline Worksheet

Establish units and the source ledger

Use decimal bandwidth consistently: 1 TB/s = 10^12 bytes/s and 1 TFLOP/s = 10^12 FLOP/s. ^[14] Binary memory units differ: 1 GiB = 2^30 bytes. ^[15] These are unit definitions used in the calculations, not performance observations. Google's explicit use of GiB for TPU memory illustrates why retaining the source unit matters. ^[12]

Table 1 defines the minimum input ledger. Its sensitivity entries are proposed checks rather than measured confidence intervals. Every row should be completed separately for the incumbent and candidate, and separately for prefill and decode.

The ledger separates supplied facts from assumptions. A large unresolved range in Q should produce a range of lower bounds, not an artificially precise latency forecast. Preserve the original unit and a normalized byte value side by side in the working spreadsheet.

Calculate the Roofline Envelope

Let Q = Q_w + Q_KV + Q_other, counting both reads and writes that reach HBM. Count work and traffic over the same interval and device boundary. NERSC requires runtime, total operations and total moved bytes for workload characterization. ^[2] The following expressions are a spreadsheet-ready calculator; seconds result when the inputs use operations, bytes and their per-second rates. ^[2]

intensity_flop_per_byte = work_flop / hbm_bytes
ridge_flop_per_byte = peak_flop_per_second / peak_hbm_bytes_per_second
compute_lower_seconds = work_flop / peak_flop_per_second
memory_lower_seconds = hbm_bytes / peak_hbm_bytes_per_second
device_lower_seconds = MAX(compute_lower_seconds, memory_lower_seconds)
roof_flop_per_second = MIN(peak_flop_per_second,
                          intensity_flop_per_byte * peak_hbm_bytes_per_second)

The maximum expresses the optimistic envelope: execution cannot take less time than either resource requirement permits. It assumes their overlap when translating the roof into time. Do not add the two terms and call that the original roofline lower bound; equally, do not assume the maximum includes launch, queuing or communication latency. ^{[1] [10]}

For a dense weight-matrix approximation, let P be the weights participating in the modeled multiplication and T the tokens processed together. Counting a multiply plus an add as two operations gives F_weight = 2PT. If those weights cross HBM once, Q_w = P b_w, so I_weight = 2T/b_w. These are analytical assumptions for the examples below, not an exact count of a complete Transformer. They implement the operation-to-traffic ratio in the original model. ^[1]

Preserve unrounded input cells and make the classification a formula, so reviewers can change assumptions directly.

For conventional single-token decode, T equals active batch B. For a simple prefill shape, T equals B times prompt length S. This explains the effect of weight reuse across tokens, but does not include attention's context-dependent work. The phase distinction is consistent with Sarathi-Serve's analysis. ^[4] If the weight-only estimate puts a workload below the ridge, include omitted traffic before drawing conclusions: additional bytes lower intensity further, whereas additional arithmetic can change the classification.

Implementation Considerations and Process Changes

Specify traffic before classifying prefill and decode

“Decode is memory bound” is a useful starting hypothesis for low active batches, not a universal label for every kernel or every serving configuration. The GQA research identifies the loading of weights and cached keys and values at each generation step as a bandwidth burden. ^[16] Larger batches can amortize weight traffic, but they also change cache requirements and the service's scheduling conditions.

Concurrency is not active batch size. The buyer should record how many requests actually contribute tokens to each modeled iteration. A request waiting in a queue does not supply useful matrix reuse. vLLM's chunked prefill can combine smaller prompt chunks with decode requests, so the observed execution shape may differ from the submitted prompt shape. ^[18]

Apply the worksheet in this order:

• Choose the interval: one decode iteration or one explicitly defined prefill chunk.
• Count active work: tokens actually scheduled, rather than requests merely connected.
• Account for weights: include packed data and associated scale information. ^[6]
• Account for attention: use current context lengths and KV-head structure. ^[16]
• Account for reuse: distinguish HBM reads avoided by cache behavior from bytes still resident. ^[13]
• Recalculate by shape: keep the distribution of bounds rather than averaging away the latency-critical requests.

The sequence is an analytical procedure, not a claim that a framework exposes all those inputs automatically. Unobserved traffic remains a labeled assumption.

Distinguish KV capacity from KV traffic

For an uncompressed cache in a conventional full-attention layout, an allocation estimate is 2 × L × B × S × H_KV × d_h × b_KV bytes. The leading factor represents keys and values. This is bookkeeping for the stated layout, not a universal traffic formula. Grouped-query attention shares a key and value head across query groups, so query-head count is not a substitute for KV-head count. ^[16]

A concrete architecture check comes from the official Qwen2.5-7B-Instruct card: it lists 28 query heads and 4 KV heads. ^[17] Substituting the query count for the KV count would inflate that factor by 7, a derived ratio. The worksheet still needs the pinned layer count, head dimension, cache datatype, current sequence lengths and actual layout before calculating bytes. Do not infer them solely from the model's name.

Allocation does not establish reads per iteration. Hugging Face documents caches that grow with generation, attention variants that cap growth, and offloading that moves cache storage to CPU memory. ^[19] Each changes the applicability of a simple full-context, HBM-only estimate. PagedAttention also allows noncontiguous KV storage, so implementation layout is part of the evidence request. ^[20]

Handle caching and quantization as changed inputs

Prefix reuse and kernel-local reuse are different mechanisms. vLLM states that automatic prefix caching reduces repeated prefill work but does not reduce the time spent generating new tokens. ^[21] Measure the prefix-hit distribution and count only the prompt work executed under the tested policy.

Weight quantization changes the representation but does not establish a matching arithmetic ceiling. TorchAO describes low-precision tensors with extra metadata, including scales and packing formats. ^[6] Request the storage datatype, runtime operand datatype and selected kernel separately. FlashInfer's attention API exposes a tensor-core option, further illustrating that an attention workload's arithmetic path is implementation-dependent. ^[22]

Limits That Can Break the Simplified Estimate

Cache hierarchy, kernel mix and overlap

An HBM roofline uses traffic crossing HBM. An L2-cache roofline uses a different boundary and byte count. NERSC describes hierarchical rooflines with different intensities at different cache levels. ^[2] Do not use an L2 traffic counter as Q beneath an HBM bandwidth roof.

Attention implementations can change Q substantially without changing the model's mathematical output. FlashAttention uses tiling to reduce memory reads and writes. ^[23] This is why an optimization or software upgrade can move the workload on the diagram rather than simply raise a hardware ceiling. Triton likewise demonstrates reuse-sensitive block ordering. ^[13]

A whole-model point can hide alternating kernel bottlenecks. Request phase-separated and major-kernel evidence in addition to the aggregate. A compatible tensor roofline remains appropriate only for the data path it describes. ^[3] Treat overlapping transfers, fused operations and kernel boundaries consistently; adding lower bounds for operations that actually overlap can overstate even the analytical floor.

Host dispatch, interconnect and runtime configuration

An HBM-only model has no term for central processing unit (CPU) launch or scheduling overhead. PyTorch documents how Compute Unified Device Architecture (CUDA) graph replay removes layers of argument setup and dispatch, with graph-safety restrictions that commonly involve static shapes and control flow. ^[7] FlashInfer documents a corresponding implementation tradeoff: graph-compatible execution can constrain kernel dispatch choices across shapes. ^[22]

Communication belongs outside a single-device roof. Ray describes tensor parallelism as splitting weights across GPUs and pipeline parallelism as splitting layers. ^[8] Ray Serve also uses locality-aware routing to reduce network latency. ^[24] A multi-device buyer must therefore request topology and communication evidence, rather than summing HBM bandwidths and treating the result as one universally accessible memory system.

Check these exclusions before treating a bound as decision-useful:

• Memory fit: include runtime reservations and working state, not nominal weights alone. ^[12]
• Launch and scheduling: establish whether the workload is device busy or host limited. ^[7]
• Communication: capture transfers and synchronization on the critical path. ^[8]
• Power configuration: preserve the tested operating conditions. ^[2]
• Kernel eligibility: verify datatype, sparsity and actual instruction path. ^[3]
• Tail latency: evaluate the declared service objective alongside aggregate throughput. ^[9]

Profiling is evidence, with collection conditions attached

Nsight Compute can require more replay passes when more metrics are selected; its default cache flushing can also change the cache state seen by repeated kernels. Host timing during collection includes profiling overhead. These are documented collection effects, so a profiler run should explain its replay and cache-control settings. ^[3]

Keep unprofiled service timing separate from the diagnostic trace. JAX provides a clear example of another timing pitfall: asynchronous dispatch can return before device work completes. (Source: docs.jax.dev) Its profiling documentation also distinguishes initial compilation and transfer costs. (Source: docs.jax.dev) A purchasing comparison should define warm-up, completion synchronization and whether cold-start costs fall inside the service target.

Hypothetical Shape Scenarios

Low-batch decode (Hypothetical Example)

Both examples use a fictional accelerator and a deliberately simplified dense model. Assume P = 8 billion participating weights, b_w = 2 bytes, C = 800 TFLOP/s and BW = 3 TB/s. Their derived ridge is 266.67 FLOP/byte. None of these figures identifies or measures a commercial GPU. The operations and traffic are explicit hypothetical inputs to the roofline equation. ^[2]

For decode, assume B = 1, one full weight read, and 0.5 GB of additional HBM traffic. Then F = 2PB = 16 GFLOP, and Q = 16.5 GB. The resulting intensity is 0.970 FLOP/byte. Compute requires at least 0.020 ms, while memory movement requires at least 5.5 ms. The larger term is the device lower bound. All values in this paragraph are derived from the stated assumptions. ^[2]

Doubling only compute leaves that 5.5 ms bound unchanged. Doubling only bandwidth changes it to 2.75 ms, assuming identical work and traffic. These are conditional model results, not expected speedups. The result is useful because it identifies the vendor evidence required: does the real execution move approximately the assumed bytes, and is device execution a material component of the service's elapsed time?

High-batch prefill (Hypothetical Example)

Now assume B = 8 prompts of 2,048 tokens processed together, giving 16,384 token positions. Hold the same fictional weights and device constant. For the weight-matrix approximation, F = 2PBS = 262.144 TFLOP. Assume 20 GB total HBM traffic, consisting of the weight read plus a hypothetical allowance for other traffic. This is a chosen traffic budget, not a measured attention implementation. ^[2]

Table 2 compares the derived quantities. The prefill excludes an explicit context-dependent attention operation count and is therefore a simplified weight-work envelope rather than a complete end-to-end prefill estimate.

Every cell is an analytical result from the hypothetical inputs and the same lower-bound formula. ^[2] The examples show why a claim about FLOPS can be plausible for one shape and irrelevant to another. They do not prove that batching achieves the assumed reuse, that attention is negligible, or that a deployed system can attain peak rates.

For reproducibility, enter the raw counts into the calculator rather than rounded intensities. For sensitivity analysis, replace Q with the optimistic and pessimistic traffic assumptions and rerun both device terms. If a conclusion reverses across that interval, classify the purchase as need more data. This is a decision rule proposed by this report, not a statistical confidence claim.

Data Analysis and Evidence

Official specifications are inputs, not benchmark outcomes

Table 3 demonstrates source interpretation across selected accelerator specifications. It is not a ranking. Figures were retrieved on October 31, 2026; an undated live page is not a guaranteed specification for every shipping configuration. The NVIDIA page retains a preliminary-specification qualification. ^[25]

The rows expose incompatible labels rather than establish a winner. A higher ridge means more work per byte is needed to reach that device's stated compute roof. It does not establish that the device is faster or slower on a given model. Vendor peaks also do not supply measured effective bandwidth for the buyer's access pattern.

Google's TPU v5p provides a unit cross-check outside the GPU rows: the documentation lists 459 TFLOPS BF16 per chip and 2,765 GB/s HBM bandwidth per chip. ^[12] Normalizing to decimal bytes gives a derived ridge of 166.00 FLOP/byte. This checks dimensional arithmetic; it is not a cross-platform inference comparison or a reason to substitute TPU software behavior into a GPU pilot.

For procurement, these ratios identify the traffic intensity needed to use a stated arithmetic ceiling. Ask the supplier to preserve the original specification row alongside each normalized value. If the proposed implementation uses another datatype or operating mode, replace that input before comparing bounds. A completed row should explain eligibility, rather than merely reproduce a larger number.

Implications and Future Directions

Request an evidence packet that can test the proposal

A useful vendor packet permits another engineer to reconstruct the inputs and compare like workloads. MLPerf Inference defines benchmarks by datasets and quality targets, as well as specified load-generation patterns. ^[27] Those principles are valuable even when the buyer's application does not match a published benchmark. SPEC ACCEL similarly emphasizes comparability and reproducibility, although its rules do not govern this proposed inference pilot. ^[28]

Request the following artifacts before authorizing a performance-based upgrade:

• Workload record: prompt and output distributions, arrival process, active batches, concurrency, prefix-hit behavior and chosen SLO.
• Model identity: exact artifact revision, weights, quantization scales, adapters and acceptance criteria for output quality.
• Hardware identity: per-device SKU, memory configuration, device count, power policy and interconnect topology.
• Software identity: driver, runtime, serving engine, kernel selection and graph-capture configuration.
• Roofline worksheet: units, input provenance, traffic assumptions, omitted work and incumbent/candidate bounds.
• Diagnostic evidence: representative phase traces and compatible kernel rooflines, with collection settings disclosed. ^[3]
• Service evidence: unprofiled TTFT and subsequent-token latency distributions, accepted-request throughput and failure accounting. Aggregate and per-user rates should be distinguished. ^[9]
• Reproducibility: disclose sufficient configuration and artifact details for another engineer to repeat the comparison.
• Economic evidence: total proposed costs and capacity at the same quality and latency target, with the accounting boundary stated.

Choose the gate outcome explicitly

Use three outcomes rather than presenting a single estimated tokens-per-second number:

• Benchmark this candidate: the relevant bound improves, workload fit is credible, and the missing implementation evidence can be obtained in a controlled pilot.
• Reject the stated claim: the proposal substitutes an incompatible compute mode, mixes device and system units, or treats a device lower bound as demonstrated service performance.
• Need more data: traffic, active batch, cache policy, communication or latency targets are too uncertain to determine whether the proposed resource matters.

A lower bound alone cannot establish a candidate’s advantage: equal-looking inputs do not establish equal utilization. Compare measured service capacity at matched quality and latency. Quail acknowledges that its ideal assumptions cannot all be satisfied. ^[11]

Measure cost after qualifying performance

Evaluate cost per accepted output token or completed request only after defining the latency and quality gates. Include the buyer's relevant acquisition or rental costs, power and operating assumptions, and the useful capacity actually delivered. No universal price or amortization period is assumed here.

Report cold-start and warmed operation separately when both affect the application. JAX documents first-execution compilation and CPU-to-accelerator transfers as relevant timing costs. (Source: docs.jax.dev) For distributed deployments, include routing and communication within the service boundary rather than attributing every improvement to GPU arithmetic. ^[24]

Conclusion

An LLM inference roofline worksheet turns a GPU upgrade proposal into a testable resource argument. Its value comes from matching arithmetic, bytes and precision to a specific phase and device boundary. A compute peak becomes meaningful only after the workload's intensity is compared with the corresponding ridge.

The buyer should keep three distinctions visible: capacity versus traffic, theoretical lower time versus observed service latency, and a resource improvement versus a validated speedup. A low-batch decode workload may provide little reuse of weights; a large prefill can provide much more. Neither label removes the need to account for attention, cache policy, scheduling and communication.

The practical deliverable is a source ledger, a range of phase-specific bounds and an evidence packet. Complete those inputs for the incumbent and candidate, label assumptions, and preserve the calculations so another engineer can reproduce them. Where a proposal changes quantization, batching or placement, recalculate the workload rather than carrying forward an old intensity.

The final purchasing gate remains empirical: compare qualified service capacity and cost under the same quality and latency requirements. Advance a candidate when the analytical opportunity and available evidence justify that test. Reject an unsupported performance claim when its inputs are incompatible. Request more data when the resource boundary is unresolved. That discipline makes the roofline a useful screen without asking it to promise a speedup it cannot establish.