Speculative Decoding Economics: When Draft Models Cut Cost

Executive Summary

Speculative decoding lets a cheap proposer guess several output tokens that the large target model then verifies in a single forward pass. The original paper reported a 2X-3X acceleration of T5-XXL with identical outputs (^[1]), and Google's retrospective states that the method guarantees an identical output distribution (Source: research.google)). Those results are real, but they were measured at batch size 1 on specific tasks, so they are observations, not a forecast for a private fleet. The decision a private-inference team faces is narrower: does the technique raise SLO-qualified accepted output per GPU-hour after paying for proposer memory, verification compute and lost batching headroom?

Published evidence says the answer depends on concurrency. NVIDIA measured Llama 3.3 70B on one H200 at batch size 1 and reported throughput speedups of 3.55x, 3.16x and 2.63x for 1B, 3B and 8B draft models (^[2]). At higher load the gain shrinks or reverses: a trained EAGLE-3 draft for Kimi K2.5 gained over 60% at batch size 1 but 26% at batch size 16 (^[3]), and vLLM's project blog reported a 1.4x slowdown for a 70B draft-model setup at high request rates (^[4]). vLLM scopes the technique to medium-to-low request rates and memory-bound workloads (^[5]). The one cost-linked vendor measurement found, from Red Hat, is a 19.4% lower cost per million output tokens on one workload (^[6]), with smaller gains on others.

A closed-form model makes the trade-off explicit. With modeled per-position acceptance probability alpha, k speculated tokens and a draft-to-target cost ratio c, the expected speedup is (1 - alpha^(k+1)) / ((1 - alpha)(kc + 1)) (^[7]). At c = 0.05 and k = 4, theoretical break-even alpha is about 0.17 and alpha = 0.7 yields 2.31x. If verification is not free, because the GPU is already near compute saturation, the same configuration at v = 0.25 of one baseline target-step time per proposed token (k*v = 1.0, or 100% extra verification time per iteration) yields 1.26x and falls below 1.0x at alpha = 0.5. The overhead term is this report's modeling assumption, not a published constant.

The recommendation is to pilot speculation where measured qualified throughput and whole-replica cost justify it, including workloads with a binding p95 inter-token latency (ITL) target. Low-to-moderate concurrency is one favorable case. Red Hat also reports throughput gains across its tested concurrency levels for the MoE model gpt-oss-120b ^[6]; MagicDec demonstrates benefits in long-context, large-batch settings ^[8]. Prefer lossless rejection-sampling variants over heuristic acceptance, and compare whole replicas at stated p95 time-to-first-token (TTFT) and ITL targets. The implementation section provides a benchmark contract, a calculator and a hypothetical worked example. Acceptance shifts with prompt domain, so one aggregate number is unsafe.

Introduction and Background

Enterprise teams running a private target model usually reach speculative decoding from a latency complaint: interactive users see slow token streams, and adding GPUs does not shorten a single request. The technique exploits idle arithmetic capacity. NVIDIA describes a small draft model that first predicts several likely next tokens (^[9]), after which the target verifies them in parallel, and IBM summarizes it as one forward pass that can produce two or more tokens for the price of one (^[10]). Google Cloud frames the payoff as generating 4-5 tokens for the memory cost of one decode step (^[11]).

The vendor and research literature mostly reports speedup ratios. Those ratios are an input to cost, not a measure of it. A private fleet pays for GPU-hours whether or not tokens are accepted, whether or not requests met their latency targets, and whether or not the extra proposer state forced a larger replica. This report therefore uses one decision metric throughout: cost per million accepted output tokens from requests that met their SLO, compared across whole replicas.

The scope is an analyst's worksheet rather than a tutorial. It separates three questions that are often blurred: whether the algorithm preserves the target's output distribution (a property of the verification rule), whether a given engine reproduces that property in practice (numerics and batching), and whether the arrangement lowers fleet cost (hardware, traffic and SLOs). Vendor figures are treated as vendor claims with stated hardware and batch size, and prices are illustrations retrieved on 2026-10-18, not quotes.

GPU Smith is an independent private AI infrastructure engineering consultancy that advises on GPU procurement, workload sizing, inference optimization and air-gapped deployments. It is not a chip manufacturer or cloud provider. No benchmark results are invented here: numerical examples in later sections are either cited, direct outputs of a cited formula, or labeled hypothetical.

Key Changes: From Tokens per Second to Cost per Accepted Token

The cost model replaces the headline speedup

Leviathan, Kalman and Matias define the cost coefficient c as the ratio between the time for a single run of the draft model and a single run of the target (^[12]), with the acceptance rate alpha measuring how well the draft approximates the target. Their expected improvement factor in walltime is (1 - alpha^(k+1)) / ((1 - alpha)(kc + 1)) (^[7]). Alpha is a property of the model pair and task; c depends on hardware and software.

The paper's experiments sat at the favorable end of the c range: the cost coefficient was always less than 0.05 (^[12]). It also states the boundary condition: the method is not helpful for configurations where additional computation resources are not available (^[12]). Check whether the production configuration has spare compute before applying the model. This report therefore adds one explicit assumption, a verification overhead v, defined as extra target-step time per speculated token. At v = 0 the formula reduces to the published one; at v = 1 verification is fully compute-bound and every speculated token costs a full token of work.

Four quantities replace one

A deployment decision needs four measured numbers:

• Observed draft acceptance fraction (r): accepted draft tokens divided by proposed draft tokens, which vLLM defines as the fraction of proposed draft tokens accepted (^[13]), sliced by prompt class and output position. This telemetry fraction counts the accepted prefix against all proposed tokens. Keep it separate from alpha, the modeled per-position acceptance probability used under the original paper's independence assumption ^[12]. Do not insert r directly into the alpha-based formula.
• Mean acceptance length: tokens emitted per verification step including the bonus token (^[13]).
• Net speedup at production concurrency, not at batch size 1.
• SLO-qualified goodput: DistServe defines per-GPU goodput as the request rate that can be served adhering to the SLO (^[14]).

Latency vocabulary should follow a documented standard. NVIDIA's benchmarking documentation defines TTFT as how long a user waits before seeing the model's output (^[15]), ITL as the average time between consecutive tokens (^[15]), and system throughput as total output token throughput across all simultaneous requests (^[15]). Speculation delivers tokens in bursts of one to k+1 per step, so averages hide the tail and percentiles should be measured client-side. Speculation also leaves the prefill stage alone: AWS found that prefill latency is neither improved nor degraded (^[16]), and Google's Vertex team notes an increase in TTFT may be observed (^[17]).

Where the extra work happens

Three costs hide inside a headline gain. First, proposer compute: the draft-to-target single-pass time ratio was about 12.5% for a 70B target and an estimated 37.5% for an 8B target in one vLLM study (^[18]), so small targets leave little room for a separate drafter. Second, verification of rejected tokens: the target still processes every proposed position. Third, scheduling: Google notes some tokens are harder and some easier to generate (Source: research.google)), so wasted verification work is uneven across a batch. Google Cloud describes the net effect as slightly increased compute costs and operational complexity, worthwhile for latency-sensitive workloads (^[11]).

Key Changes: Lossless Sampling Versus Heuristic Acceptance

What the guarantee covers

Chen et al. at DeepMind introduced speculative sampling as a modified rejection sampling scheme which preserves the distribution of the target model (^[19]), and a later survey summarizes that both original papers theoretically proved identical output distributions to the target (^[20]). Engine documentation echoes this: vLLM Speculators states that accepted tokens are guaranteed to come from the same distribution (^[21]), and Hugging Face says the verification forward pass ensures the same outputs are generated as if the large model had run alone (^[22]).

The guarantee has a stated scope. vLLM calls speculative sampling theoretically lossless up to the precision limits of hardware numerics (^[5]) and says its implementation is algorithmically validated to be lossless, yet it does not guarantee stable token log probabilities (^[5]), and changes in batch size may cause variations in logprobs and output probabilities (^[5]). The practical reading: speculative and non-speculative runs are distributionally equivalent under the algorithm, not byte-identical.

Heuristic variants change the distribution

Several methods trade exactness for acceptance, so an equivalence claim must be checked per method:

• Lenient acceptance: the original paper's appendix describes a lenience parameter; on T5-XXL it produced improvement factors of 2.5X, 3.1X, 3.6X and 5X as leniency increased (^[7]), bought by altering the output distribution.
• Medusa typical acceptance: the authors write that they do not insist on an exact correspondence between the output and language model distribution (^[23]).
• Exact variants: EAGLE states that preservation of the output distribution is theoretically guaranteed (^[24]).
• Cross-tokenizer drafts: vLLM's token-level intersection mode allows draft models from a different model family with a different tokenizer (^[25]), but probabilistic draft sampling is not yet supported in that mode (^[25]), so check the sampling configuration before assuming equivalence.

An equivalence test that survives numerics

Run-to-run differences exist even without speculation. Thinking Machines attributes nondeterminism in nearly all inference endpoints to varying load (^[26]) and explains that decreasing batch size eventually leads to more cores than batch elements, changing reduction strategies (^[26]). vLLM does not guarantee reproducibility by default (^[27]) and provides it only on the same hardware (^[27]). A batched speculative decoding study verified equivalence with deterministic greedy decoding to eliminate sampling variance (^[28]) and attributed residual mismatch to floating-point non-determinism.

Keep the repeated non-speculative baseline as a numerical-stability control under matched hardware, engine configuration and load. Use exact-match comparisons separately for deterministic greedy decoding ^[28]. For stochastic sampling, use a pre-specified distribution test with enough independent samples to detect a declared discrepancy at a declared significance and power; vLLM separately validates rejection-sampler convergence to the target distribution ^[5]. Add a held-out task-quality gate for both lossless and lossy methods. Do not treat two-run exact match or aggregate quality parity as establishing stochastic distributional equivalence.

Key Changes: Memory, Compute and Strategy Choice

Memory is paid twice

The target's weights and KV cache are unchanged and the proposer adds its own. vLLM's project blog notes the memory manager now handles the KV cache for both the draft and target models (^[4]), and NVIDIA's NIM documentation lists a separate draft model with its own weights and KV cache (^[29]). IBM's speculator design avoids duplicating the full KV-cache across candidates (^[10]). In one vLLM study, per-token KV cache rose from 144 KiB to 256 KiB for Qwen3-8B with a 0.6B draft, and from 320 KiB to 352 KiB for Llama3-70B with a 1B draft (^[18]). On smaller accelerators the weights alone matter: an AWS walkthrough with a 70B target and 7B draft found the two models need almost 200 GB of device memory for the weights (^[30]). BentoML warns that on a single GPU the draft can quickly squeeze out space for other tasks such as batching (^[31]).

A worked calculation (Hypothetical Example; the arithmetic and the 65 GB assumption are this report's, the per-token figures are the cited ones) shows the effect. Take a 70B model in 8-bit weights on one H200, which NVIDIA lists at 141 GB of HBM3e at 4.8 TB/s (^[32]). Assume 65 GB remains for KV cache before a roughly 2.5 GB, 1B-parameter FP16 draft is loaded. At 320 KiB per token the pool holds about 198,000 tokens. With the draft, 62.5 GB at 352 KiB per token holds about 173,000 tokens, a 12.6% reduction in concurrent-context capacity. Whether that matters depends on how close the fleet runs to its KV limit.

Strategy matrix

Table 1 below summarizes candidate strategies on the dimensions that drive cost: extra state, output equivalence and evidence. Speedups are the source's own, on the source's hardware, and are not comparable across rows.

The table shows that "extra state" ranges from nothing (n-gram) to a second full model, and that the strongest evidence for each method sits at batch size 1 or in a vendor's own harness. vLLM's guidance is that EAGLE, MTP, draft models and PARD provide the best latency reduction (^[5]), but also that real gains depend on model family, traffic pattern, hardware and sampling settings.

Selecting a draft or speculator

Selection should be an ordered filter, not a leaderboard:

• Tokenizer and vocabulary: Standard Hugging Face assisted generation uses a shared tokenizer (^[22]); Universal Assisted Generation supports different tokenizers ^[40]. Benchmark the chosen mode, including token translation and its sampling constraints, and TensorRT-LLM warns that a mismatch makes acceptance extremely low and performance regressed (^[41]).
• Size ratio: Hugging Face reports meaningful speedups are typically seen when the assistant is at least 50-100 times smaller (^[40]); Baseten's guide says at least ten times smaller by parameter count (^[42]). These rules of thumb differ by 5x, so measure.
• Smaller is not always better: in AWS's Trainium test a 0.6B draft was faster but had roughly 60% lower acceptance, cancelling the savings, and the 1.7B draft with 7 speculative tokens gave the best balance (^[16]) (^[16]). Together AI states the same trade-off: larger speculators yield higher acceptance rates due to their capacity (^[43]), but draft more slowly.
• Trained speculators and their cost: Red Hat reports two to three times the throughput gains for trained speculators (^[44]), and Together reports a trained speculator lifting Kimi from about 150 to over 270 tokens per second (^[43]). Training needs internal hidden states from the target model (^[21]); a Kimi K2.5 EAGLE-3 draft took 1500 H200 GPU-hours across 600,000 training samples and 6 billion tokens (^[3]). Snowflake trained its gpt-oss speculators for 3,000 steps (approximately one epoch) on Ultrachat and Magicoder using 16 H200 GPUs: three elapsed hours for gpt-oss-20B and 21 elapsed hours for gpt-oss-120B, equivalent to 48 and 336 H200 GPU-hours, respectively (16 GPUs multiplied by elapsed hours) (^[45]). These reports concern different target models and training workloads; do not treat the figures as a controlled training-efficiency comparison.
• Engine support: TensorRT-LLM's EAGLE 3 mode does not support tree structures for draft sequences (^[41]), so the same method name behaves differently across engines.
• Context length: BentoML found out-of-the-box EAGLE 3 heads were not optimized for long contexts (^[46]).
• Alignment: DistillSpec uses knowledge distillation to better align the draft with the target.

Key Changes: Concurrency Moves the Break-Even

Why the sign of the benefit depends on load

Speculation spends spare arithmetic to save memory-bandwidth-bound steps. NVIDIA's NIM documentation says it helps most when the GPU has spare compute, for example at low-to-moderate concurrency (^[29]), and warns that under saturated load it can fail to improve throughput, and can reduce it (^[29]). NVIDIA's co-design analysis quantifies one side of this: with a draft length of seven, one-eighth of the batch size is needed to become compute bound compared with no speculation (^[9]). Baseten's guide states the engineering consequence: speculation must be dynamically disabled as compute is too saturated to afford verification (^[42]), and it adds that adopting EAGLE for lower latency requires reduced batch sizes, lowering throughput and increasing cost (^[42]).

Measured degradation with load

The measurements agree on direction and differ on the threshold. AMD's MI300X study with a 34B target and a 1.1B draft found graph-mode slowdown from batch size 32 (^[47]). PyTorch and IBM observed throughput reduction beyond a batch size of 64 for 7B to 20B models with speculator heads (^[48]). vLLM's project blog measured a 1.5x speedup at QPS 1 for Llama3-70B on four H100s (^[4]) but a 1.4x slowdown at high QPS (^[4]). BentoML's informal H100 tests on one GPU found total throughput plateaued earlier than the baseline (^[31]); it labels these results as informal and for reference only (^[31]). Snowflake saw a CPU-side method become worse at a concurrency of 64 before optimization (^[49]).

Research explains the mechanism: as the system becomes compute-bound, the overhead spent proposing and verifying rejected tokens becomes more costly (^[18]), which is why closed-loop controllers that disable speculation under load have been proposed. TensorRT-LLM documents the static-configuration consequence: speedups are only observable at low batch sizes (^[41]). Gains are not uniformly fragile, though. AMD's 405B study found the speedup declined only slightly with increasing request rates (^[50]), which shows the threshold depends on the model, hardware and engine.

The long-context exception

MagicDec provides a counterweight: for moderate-to-long sequences, the cost of loading the KV cache dominates as batch size grows, so the high-batch limitation only applies to very short sequences (^[8]). It reports up to 2.51x speedup for LLaMA-3.1-8B at batch sizes 32 to 256 (^[8]). AMD likewise found speculation can boost performance further when the input sequence is longer (^[50]). The region where speculation helps at scale is long-context, KV-bound decode on high-FLOPS hardware, which differs from short-prompt chat at moderate load.

Implementation Considerations and Process Changes

Benchmark contract

A pilot is comparable only if both arms obey the same contract. The fields below belong in the results template.

• Models: target and draft or speculator checkpoint hashes, tokenizer hash, quantization format.
• Software: engine name and version, speculative method, and speculative-token setting (vLLM's num_speculative_tokens is the number of tokens proposed per step (^[5])), sampling parameters.
• Hardware: GPU model and count, interconnect, driver and CUDA version, tensor-parallel degree.
• Load: closed-loop concurrency levels or open-loop arrival rates, prompt and output length distributions, prompt-class mix.
• Latency definitions: TTFT, ITL and end-to-end latency at p50, p95 and p99, with the SLO stated before the run.
• Throughput definition: NVIDIA's own throughput figure is inclusive of time to generate the first token (^[2]), so state whether prefill is included.
• Equivalence: baseline numerical-stability repeat, deterministic greedy comparison, stochastic distribution-test design and result (including sample size, significance and power), and held-out quality-gate outcome.

The baseline is the same target, engine and hardware with speculation disabled, tuned to its own best batch limit. NVIDIA's documentation advises measuring acceptance and throughput on representative traffic before committing to a method (^[29]).

Telemetry and slices

Record vLLM's acceptance rate and mean acceptance length, plus rejected-token work, sliced by:

• Prompt class: code, extraction, long-form generation, multi-turn chat, retrieval-grounded answers. AWS found open-ended prompts gave no consistent benefit while structured, repetitive prompts did (^[16]).
• Output position: acceptance usually decays with draft depth, so an average hides which depth to cut.
• Sampling mode: Chen et al. observed the acceptance rate depends on the application and the decoding method (^[51]).
• Concurrency bucket: net speedup at 1, 4, 16 and 64 concurrent requests, not only the peak.

Domain effects are large. A domain-specific draft study finds the acceptance rate of the generic draft model drops significantly due to domain shift (^[52]). Fireworks reported a generic draft model increasing latency by 1.5x on a specialized customer workload (^[53]). Snowflake reports acceptance averaging 44%-50% for its gpt-oss speculators (^[45]) and notes the best speculator balances acceptance against computational resources for the specific model and workload (^[54]).

Results template and calculator

The idealized speedup grid appears as Table 3 in the Data Analysis section. The fleet-level calculation that matters for budgeting follows this chain: measure SLO-qualified accepted tokens per second per replica (excluding rejected speculative work and SLO-missing requests), divide peak demand by it, round up to whole replicas, multiply by GPUs per replica and the GPU-hour price, and divide by tokens produced. A minimal calculator:

from math import ceil

def tokens_per_step(alpha_per_position, k):
    # Theoretical worksheet: independent per-position acceptance.
    return sum(alpha_per_position**i for i in range(k + 1))

def net_speedup(alpha_per_position, k, c, v=0.0):
    # c: draft step / baseline target step.
    # v: extra verification time per proposed token / baseline target step.
    return tokens_per_step(alpha_per_position, k) / (1 + k * (c + v))

def measured_decode_speedup(mean_acceptance_length, step_cost_ratio):
    # step_cost_ratio: measured speculative iteration time / baseline decode-step time.
    return mean_acceptance_length / step_cost_ratio

# Fixed-k worksheet algebra, not a fitted production threshold:
# L = 1 + k*r; modeled step-cost ratio = 1 + k*(c+v).
# Break-even: L > 1 + k*(c+v), equivalently r > c+v.
# At k=4, c=0.05, v=0: L > 1.2 and r > 0.05.
# The separate theoretical alpha threshold is approximately 0.17.

def cost_per_million(gpus_per_replica, gpu_hour_usd,
                     qualified_tokens_per_s, demand_tokens_per_s,
                     actual_qualified_fleet_tokens_per_s):
    if min(gpus_per_replica, gpu_hour_usd, qualified_tokens_per_s,
           demand_tokens_per_s, actual_qualified_fleet_tokens_per_s) <= 0:
        raise ValueError("All inputs must be positive")
    replicas = ceil(demand_tokens_per_s / qualified_tokens_per_s)
    capacity = replicas * qualified_tokens_per_s
    if actual_qualified_fleet_tokens_per_s > min(demand_tokens_per_s, capacity):
        raise ValueError("Served qualified throughput exceeds demand or capacity")
    fleet_cost = (replicas * gpus_per_replica * gpu_hour_usd * 1e6
                  / (3600 * actual_qualified_fleet_tokens_per_s))
    return replicas, fleet_cost

For the price input, list prices vary widely. As retrieved on 2026-10-18, CoreWeave's undated pricing page listed an 8-GPU H100 node at $49.24 per hour on demand (^[55]) and an 8-GPU H200 node at $50.44 per hour (^[55]), roughly $6.16 and $6.31 per GPU-hour. Use the organization's actual fully loaded rate, including power and cooling for owned hardware. Fireworks makes the broader point that a volume-optimized 70B configuration on eight GPUs can be 4x cheaper per token than one tuned for single-request speed (^[53]), the regime where speculation typically helps.

Worked example (Hypothetical Example)

Table 2 applies the calculator to invented numbers to show the mechanics. Assume 2 GPUs per replica at $4.00 per GPU-hour ($8.00 per replica-hour) and a peak demand of 20,000 SLO-qualified tokens per second. None of the throughput figures below is a measurement.

Scenario C is where speculation earns its keep: a tight ITL target forces the baseline to run small batches with poor utilization, so spare compute exists. Scenario B is the negative case: a heavier proposer costs more than doing nothing. Rounding to whole replicas matters: in scenario A the 7-replica figure comes from 20,000 / 3,125 = 6.4, so the realized fleet cost per million tokens is $0.78 rather than $0.711. The same rounding moves scenario B to $1.00 and scenario C to $1.67. Energy should be added where wall-power measurements exist. A published study cautions that lower inference time does not always correlate with proportional energy savings (^[56]), though its measurements were taken at batch size 1 (^[57]).

Production gates, drift monitoring and rollback

• Entry gate: net SLO-qualified cost per million accepted tokens lower than baseline by more than the measurement noise, at the production concurrency distribution, on at least three prompt classes.
• Canary: route a small share of traffic to a speculative replica and compare acceptance, ITL percentiles and quality-gate scores with a matched baseline replica.
• Drift monitors: track rolling mean acceptance length, including the bonus token, for each prompt class ^[13]. Compare it with the measured speculative-iteration/baseline-step cost ratio using measured_decode_speedup; alert when that ratio no longer shows a decode benefit, and recheck whole-fleet SLO-qualified cost. Keep the theoretical alpha threshold separate from the observed accepted/proposed fraction.
• Revalidate after fine-tuning, tokenizer changes, engine upgrades, a draft or speculator swap, or a traffic-mix change. Online adaptation of a draft to the query distribution raised token acceptance by 0.1 to 0.65 in one study, which cuts both ways for drift.
• Rollback: keep a speculation-disabled configuration deployable by flag; TensorRT-LLM documents that speculation cannot be disabled dynamically in its mode (^[41]), so plan rollback there as a replica-level swap.

Negative Results: When Speculation Is Not Worth It

The following conditions have published support as cases where speculation yields little or nothing. None is a flaw of a specific product; each follows from the arithmetic above.

• High concurrency on short contexts: one study measured EAGLE's speedup falling from 1.73x at batch size 1 to 1.21x at batch size 128 while absolute throughput rose (^[18]). A 1.21x gain with a 12.6% KV capacity loss can net out to zero.
• Naive static configuration under load: a serving study measured a 14% latency increase from even one speculated token at high request rates (^[58]).
• Tokenizer mismatch or poorly aligned drafts: acceptance extremely low and performance regressed (^[41]). Fireworks' generic-draft latency penalty of 1.5x is the same failure on a specialized workload (^[53]).
• Draft cost too high relative to the target: with c = 0.10 and alpha = 0.5 the formula's best case is only 1.46x, at k = 2.
• Open-ended generation: AWS saw no consistent benefit on open-ended prompts in its own test.
• Over-deep speculation: SGLang's documentation notes that increasing speculation range risks rejection cascades (^[59]), and wider candidate trees increase memory and compute consumption (^[59]) and GPU memory usage (^[59]); reducing them frees headroom for activations and CUDA graph buffers (^[59]).
• Resource contention: Baseten found draft and target models contending on one GPU ran at half speed until execution was synchronized (^[60]).
• Unsupported serving modes: Hugging Face assisted generation does not support batched inputs (^[22]) and launched limited to a batch size of one (^[61]), so it is unsuited to multi-tenant serving.
• No whole-fleet saving: if speculation does not lower SLO-qualified whole-fleet cost after accounting for its memory footprint, reject the configuration. An already-met SLO alone is not a rejection criterion; scenario A illustrates the worksheet's loose-ITL case.
• Energy-constrained sites: proportional energy savings are not guaranteed.

A negative result should be published with the same detail as a positive one: engine version, hardware, concurrency, prompt class and measured acceptance. A table of failing cells is the most useful output of a pilot because it marks the edge of the operating region.

Data Analysis and Evidence

Idealized speedup grid

Table 3 below is computed directly from the published formula with c = 0.05, the upper edge of the draft-cost range in the original experiments, and v = 0. The right-hand columns show the k = 4 configuration under two stress assumptions of this report: c = 0.10, and v = 0.25 of one baseline target-step time per proposed token. At k = 4, the latter adds k*v = 1.0 baseline target-step time per iteration: 100% extra verification time, in addition to the 20% draft cost.

Three readings follow. First, the optimum k rises with acceptance: at alpha = 0.5 the grid peaks at k = 3, while at 0.9 it is still rising at k = 7, so one global speculative-token setting is a compromise across prompt classes. Second, doubling c from 0.05 to 0.10 costs 0.2x to 0.5x at k = 4, so a draft at 10% of the target's step time is materially less valuable than one at 5%. Third, v = 0.25 per proposed token (100% extra verification time per iteration at k = 4) removes about 0.7x to 1.6x everywhere and turns alpha = 0.5 into a net loss. Theoretical break-even per-position alpha at k = 4 and c = 0.05 rises from 0.17 at v = 0 to 0.38 at v = 0.1, 0.58 at v = 0.25 and 0.78 at v = 0.5, all computed from the same formula. The formula assumes independent per-position acceptances; do not substitute the aggregate accepted/proposed telemetry fraction for alpha. Use the calculator's measured acceptance-length path for operational comparisons.

Published measurements, with context

• Original task observations: 2.6X on T5-XXL English-German translation at temperature 1; the 2X-3X range is an observation on those tasks, not a forecast.
• Batch size 1, vendor harness: 3.55x for Llama 3.3 70B with a 1B draft on one H200 (^[2]); NVIDIA's earlier post headlines up to 3.6x (^[62]).
• Draft pair dependence: PyTorch notes runtime performance varies depending on the generated text (^[33]), and its Llama-7B with a small draft pair obtained about a 1.3x boost (^[33]).
• Production claims: IBM reports cutting the latency of serving its Granite 20B code model in half while quadrupling throughput (^[10]), IBM cites latency lowered by two to three times (^[10]), and PyTorch and IBM report nearly twice as fast for Llama2 13B chat (^[48]). These are vendor claims on vendor configurations.
• Cost-linked: Red Hat's gpt-oss-120b study reports peak output throughput of 2574 tokens/s on ShareGPT (^[6]) and a 19.4% reduction in cost per million output tokens on one workload (^[6]); the cost figure is workload-specific.
• Low-load results: Snowflake's Llama 3.1 70B benchmark used an arrival rate of 0.5 requests per second (^[54]), so its speedups are low-load results.
• Task variation: a benchmark survey defines six subtasks including translation and summarization (^[20]), evaluated on Vicuna-7B at a batch size of 1.
• Production adoption: Google states speculative decoding enables AI Overviews in Google Search to produce results faster than before (Source: research.google)), which shows deployment viability, not private-fleet economics.
• Documentation is not evidence: vLLM Speculators describes the technique as speeding up inference without changing the output (^[21]) and shows how to serve a pre-trained speculator (^[21]), but it documents deployment and does not establish economics for a specific model pair.

The evidence is thick at batch size 1 and thin at production concurrency with SLO-qualified accounting. No public source reviewed provides a fleet break-even for a private target, which is the gap the worksheet in this report is designed to fill.

Implications and Future Directions

For procurement and capacity planning, compare measured SLO-qualified throughput, proposer footprint and whole-fleet cost at the relevant load. A binding ITL target is one favorable use case, as scenario C illustrates; scenario A also reduces the fleet with a loose ITL target. Include the high-concurrency MoE results discussed above and MagicDec's long-context, large-batch results in the pilot matrix ^[8]. GPU Smith, as an independent advisor on inference optimization and workload sizing, would frame a pilot as a sizing question first and an optimization second, on a vendor-neutral measurement contract.

Memory headroom becomes a first-class input. On GPUs with large HBM, such as the H200 at 141 GB (^[32]), the draft footprint is easier to absorb; on smaller parts the 12.6% KV capacity loss in the worked example can force a larger tensor-parallel degree and erase the gain.

Three developments deserve monitoring as of October 2026. Trained speculators are becoming packaged artifacts: vLLM publishes EAGLE draft models on the Hugging Face hub (^[63]), which lowers the cost of trying one. MTP-style heads ship with some models, but vLLM supports them only for families that include them (^[39]), and its MTP drafter can hold a quantized copy of the output head whose memory must be budgeted (^[39]). Concurrency-based switching is already documented in vLLM through num_speculative_tokens_per_batch_size: its example uses K=3, K=1 and K=0 at increasing concurrency, with K=0 producing no drafts ^[64]. Pin and validate the installed version: the documented examples set VLLM_USE_V2_MODEL_RUNNER=0, the page distinguishes CUDA-graph support by runner, and the option is incompatible with data parallelism. This configured concurrency policy is separate from automatic feedback-driven tuning. Keep the earlier TensorRT-LLM 1.3.0rc7 rollback limitation scoped to that documented mode and version.

The measurement gap is the largest open item. Energy per accepted token on data-center GPUs at production batch sizes was not found in the sources reviewed, and the only energy study located used workstation-class GPUs at batch size 1. Teams with wall-power meters can contribute the most useful data by publishing it.

Frequently Asked Questions (FAQs)

Does speculative decoding reduce inference cost?

It can, but not by default. Cost falls when net speedup at production concurrency exceeds the added GPU footprint and the pilot meets its SLO. Red Hat measured a 19.4% lower cost per million output tokens on one workload (^[6]), while vLLM's project blog measured a slowdown at high request rates (^[4]).

How much speedup should be expected from a given acceptance rate?

For the modeled per-position alpha, use the published formula: at c = 0.05 and k = 4, alpha = 0.6 gives 1.92x and alpha = 0.8 gives 2.80x in Table 3, before the report's incremental verification term. For observed telemetry, use the calculator's measured mean acceptance length and step-cost ratio instead. The fixed-k worksheet gives r > c+v, or r > 0.05 for c = 0.05 and v = 0; approximately 0.17 is the separate modeled alpha threshold.

How should a draft model be selected?

Check tokenizer requirements for the selected engine mode; the standard Hugging Face assisted-generation path uses a shared tokenizer to avoid token re-encoding (^[22]), prefer the smallest draft that holds acceptance (^[16]), and consider trained speculators (^[43]). Measure acceptance on the actual prompt classes, since acceptance depends on the application and decoding method.

Where does speculation sit among private LLM inference cost optimizations?

It is a latency lever that can become a cost lever. Baseten notes that adopting EAGLE for lower latency requires reduced batch sizes, lowering throughput and increasing cost (^[42]), while Fireworks observes that a volume-optimized 70B configuration on eight GPUs can be 4x cheaper per token than one tuned for single-request speed (^[53]). Compare speculation against batching, parallelism and capacity changes on the same SLO-qualified metric before committing.

Is speculative decoding better than increasing batch size?

They compete for the same spare compute. NVIDIA's analysis shows speculation reaches the compute-bound point at a much smaller batch (^[9]), and vLLM scopes the method to medium-to-low request rates (^[5]). At saturation, batching generally wins; at tight ITL targets, speculation can win.

What is the cost per token for a self-hosted model with speculation?

Divide fully loaded GPU-hours by accepted output tokens from SLO-meeting requests, then multiply by the GPU-hour price and one million, on whole replicas. The calculator and Table 2 show the arithmetic.

When is it not worth it?

At high concurrency on short contexts, with mismatched tokenizers, with a heavy draft, on open-ended prompts, or when draft memory reduces KV capacity enough to force a larger replica (see the negative-results section).

Does it change the model's outputs?

Under rejection-sampling verification the distribution is the target's, subject to hardware numerics (^[5]); heuristic acceptance schemes do not preserve it.

Conclusion

Speculative decoding is a conditional cost reduction, not a free speedup. The published algorithm guarantees the target distribution under rejection sampling, the original experiments show 2X-3X gains at batch size 1, and vendor measurements show similar gains in low-concurrency settings. The same sources and several engineering reports also show the gain shrinking as load rises, reversing for short contexts at high concurrency, and costing memory that the target may need for its KV cache.

The decision rule that follows is to measure SLO-qualified accepted tokens per second per whole replica, with and without speculation, at the production concurrency and prompt mix, and to adopt only where cost per million accepted tokens falls by more than the measurement noise. Pilot a lossless, trained speculator first where a tight p95 ITL binds, keep n-gram methods for input-grounded workloads, slice acceptance by prompt class and output position, and publish negative cells alongside positive ones. Revalidate whenever the model, tokenizer, engine, speculator or traffic mix changes, because acceptance is a property of the workload and not of the technique.