MLPerf Inference 6.1: A GPU Procurement Guide

Executive Summary

MLPerf Inference 6.1, published on September 16, 2026, is useful for GPU procurement when treated as a structured evidence set, not a universal leaderboard. The release covers 10 datacenter and 6 edge benchmarks ^[1] and 120 systems ^[2]. It adds end-to-end retrieval-augmented generation (RAG), an edge agentic test, and permitted speculative decoding for GPT-OSS-120B Interactive ^[3]. More than half of submitters used the API-centric harness ^[4]. These changes improve relevance, but they also create more ways to compare unlike rows.

The procurement rule is simple: compare only a cohort with the same release, suite, division, benchmark, accuracy target, scenario, availability class, unit, and applicable quality or latency constraint. Closed is the foundation for like-for-like comparison because its model must remain mathematically equivalent ^[5]. Open can disclose useful techniques, but it permits arbitrary preprocessing, postprocessing, and model choices ^[6]. Likewise, Offline sends all samples in one query ^[7], while Server and Interactive model Poisson request arrivals ^[8]. Their outputs answer different purchasing questions.

The worked scale example illustrates the discipline. A Closed, available Qwen3-VL Offline row reports 58.395875 samples/s on four GB200 accelerators ^[9]; the 16-accelerator family member reports 230.088793 samples/s ^[10]. That yields 98.50% descriptive scale efficiency, but it does not prove that any buyer will reproduce the result or that one GPU will deliver the quotient.

MLPerf does not provide price, lead time, production utilization, energy tariff, or facility cost. Buyers should attach dated quotes, metered power, support scope, and their own traffic trace. The v6.1 summary contains no power result for the sampled rows, so GPU thermal design power must not be substituted for measured system energy. A shortlist should therefore end in a buyer-owned pilot: clone exact artifacts, freeze hardware and software, run accuracy and performance, then replay the buyer's prompts and service-level objective (SLO). GPU Smith's documented approach similarly derives topology and node count from workload and throughput requirements ^[11] and validates against written acceptance criteria ^[12].

Introduction and Background

An MLPerf result is a measurement of a declared system under a defined workload. It is not a GPU specification, a price sheet, or a forecast of an application's service level. MLCommons describes the suite as designed to measure machine-learning model performance ^[13], while the public v6.1 repository contains both results and submitted code ^[14]. The distinction matters for enterprise AI infrastructure buyers, platform owners, performance engineers, finance teams, and investors evaluating private inference.

The correct first question is not, "Which accelerator won?" It is, "Which benchmark and scenario resemble the service being purchased?" A batch pipeline with a full queue may map to Offline. A request-serving endpoint may map to Server. An interactive large-language-model (LLM) service requires attention to time to first token (TTFT), time per output token (TPOT), and throughput together. MLPerf separately collects TTFT for the first generated token ^[15]. A multi-stage knowledge system may map more closely to the new RAG workload, which reports documents per second for ingestion and tasks per second for question answering ^[16].

Where none of these models, request distributions, context lengths, quality targets, or latency limits resembles the intended service, the benchmark cannot justify a hardware conclusion. It can still identify systems for a lab pilot. This report provides the worksheet that connects the public evidence to that pilot without treating cross-row ratios as procurement facts.

Key Changes in MLPerf Inference 6.1

New end-to-end and agentic workloads

The end-to-end RAG test measures a pipeline of multiple models for question answering ^[17]. It includes embedding, retrieval, reranking, and LLM reasoning. It exposes two independent workloads: corpus ingestion and question answering against a prebuilt vector database ^[18]. The first release focuses on Offline rather than Server ^[19]. Consequently, it informs throughput sizing for saturated pipelines but does not directly establish an online RAG latency SLO.

The Edge Agentic benchmark moves from single-shot inference toward multi-turn work ^[20]. It combines an edge model, quantization, and single-stream coding workload ^[21]. Buyers should not transfer an edge single-stream result to a datacenter concurrency forecast.

Optimization and harness changes

GPT-OSS-120B Interactive now permits speculative decoding, but the rule is not an unrestricted license. Speculative decoding is allowed only for specific benchmark and scenario combinations ^[22], and implementations must use the reference multi-token prediction head ^[23]. A vendor percentage that mixes a speculative Interactive entry with a non-speculative or different-scenario baseline is not a valid procurement ratio.

The API-centric endpoint harness uses a client/server architecture over standard APIs ^[24]. Its artifacts differ from traditional LoadGen submissions: endpoint results use structured JSON and YAML files ^[25], including result_summary.json and accuracy_results.json ^[26]. Procurement reviewers should therefore expect different filenames without treating that difference as missing evidence.

New hardware and availability labels

MLCommons classifies AMD Instinct MI350P and Intel Arc Pro B70 as available, while NVIDIA Rubin and Vera Rubin NVL72 are preview in this round ^{[27] [28]}. In MLPerf terminology, Available means all components can be purchased or rented ^[29]; Preview means expected availability in the next submission round ^[30]. It is a benchmark classification, not a quotation, allocation, support term, or promised delivery date.

Building a Valid Comparison Cohort

Every candidate row should pass the same filter before arithmetic begins. Table 1 is the minimum comparable-cohort worksheet.

The table prevents a common error: sorting visually adjacent results that answer different questions. MLCommons supports Offline, Server, Interactive, SingleStream, and MultiStream scenarios ^[31]. A comparison is defensible only after every row passes every relevant field.

Closed versus Open

Closed permits calibration for quantization but no retraining ^[32]. This makes it the preferred division for a procurement ranking, while still requiring inspection of precision, kernels, runtime, and scenario settings. Open reports achieved accuracy rather than Closed accuracy constraints ^[33]. It can reveal techniques worth testing, but it must remain a separate analytical cohort.

Lambda's v6.1 explanation is a useful illustration: its Open agentic entry kept the reference workload but swapped the served model ^[34]. Lambda expressly says the result is not an apples-to-apples comparison ^[35]. That is an optimization lead, not a price-performance ranking input.

Server, Offline, and Interactive

• Offline: asks how quickly a fully available queue can be completed. It favors sustained batch throughput.

• Server: tests a target query rate under random arrivals and applicable latency constraints. It is closer to concurrent request serving.

• Interactive: adds user-visible generation constraints. It should be assessed through TTFT, TPOT, and throughput, not throughput alone.

• Do not substitute: Offline throughput for Server capacity, or aggregate throughput for user-visible responsiveness.

• Map the trace: compare the benchmark's input and output lengths, arrival pattern, concurrency, and quality threshold with the buyer's traffic.

Server throughput should normally be expected to differ from Offline throughput because the request process and latency obligations change. This is not inefficiency in the colloquial sense. It is the cost of satisfying a different service condition.

Reading the System, Not Just the Accelerator

The repository's system-description JSON combines hardware and software information ^[36]. Each shortlisted row should be expanded into an evidence record:

• Identity: submitter, system name, repository result path, division, suite, availability, and validity.

• Workload: benchmark, scenario, accuracy target, result, unit, and latency or quality constraints.

• Accelerators: exact model, count, memory, interconnect, and configured power settings.

• Host: CPU model and count, DRAM capacity, NUMA layout, storage, and operating system.

• Fabric: network interface, link rate, switch topology, and whether traffic crosses nodes or regions.

• Software: driver, runtime, framework, inference server, quantization, kernel, model commit, and harness.

• Thermal design: air or liquid cooling, rack density, facility water and power assumptions, and measured run conditions.

• Artifacts: summary row, system JSON, measurements, config, code, accuracy output, performance output, audit files, and supplemental statement.

Hardware specifications show why the system boundary matters. AMD lists MI350P with 600 W maximum board power, configurable to 450 W ^[37], while its v6.1 technical material lists 144 GB HBM3E and 4 TB/s bandwidth ^[38]. The same source describes the part as a PCIe 5.0 dual-slot card ^[39]. These are component specifications, not measured system energy or application throughput.

At rack scale, NVIDIA's GB300 NVL72 reference architecture is liquid cooled ^[40], specifies 130 TB/s aggregate NVSwitch bandwidth ^[41], and can require up to 142 kW ^[42]. Its ConnectX-8 design offers up to 800 Gb/s per SuperNIC ^[43]. A per-accelerator quotient that omits this topology and facility context is descriptive bookkeeping, not a deployable design.

OEM configuration is equally material. Dell documents its 10U PowerEdge XE9785 with eight B300 SXM6 GPUs at 270 GB and 1,100 W each ^[44], while Supermicro lists a direct-liquid-cooled four-rack-unit system with eight MI355X accelerators ^[45]. HPE separately lists an MI350P PCIe accelerator in its portfolio ^[46]. Intel announced Arc Pro B70 availability beginning March 25, 2026 ^[47]. These facts change rack count, cooling plant, and bill of materials even when the GPU label appears similar.

Software creates another boundary. Dell disclosed TensorRT 10.14, CUDA 13.1, cuDNN 9.17, and TensorRT-LLM for B300 submissions ^[48]. Dell also describes an MI350P recipe using MXFP4 W4A4 weights and an FP8 key-value cache ^[49]. Intel identifies PyTorch vLLM on Ubuntu 24.04 for its Arc Pro B70 disclosure ^[50], and reports a four-GPU B70 node with 128 GB of VRAM ^[51]. Neither stack should be assumed portable to another vendor or release.

Implementation Considerations and Process Changes

Trace a headline to artifacts

A procurement analyst should be able to walk from a slide to a public result without relying on the slide's arithmetic:

1. Record the claim exactly. Capture the stated ratio, benchmark, scenario, system scale, and whether MLCommons verified it.

2. Locate the summary row. Confirm submitter, system name, division, availability, benchmark, scenario, accuracy, result, and unit.

3. Open the system JSON. Verify accelerator count, CPU, memory, node count, network, storage, cooling, and software.

4. Open the configuration. Confirm precision, quantization, batch controls, speculative decoding, TTFT, TPOT, and target query rate.

5. Open measurements and logs. Confirm the performance run, accuracy run, valid flag, audit output, and any power workflow.

6. Inspect code and README. Determine whether the artifacts are sufficient to build and reproduce the implementation.

7. Read supplemental material cautiously. Treat submitter statements as context, not replacements for audited fields.

8. Freeze the cohort. Only after all fields match should ratios or normalized outputs be calculated.

MLPerf requires one LoadGen validation run for each submitted performance result ^[52], and accuracy and performance modes must execute the same code ^[53]. Submission source must be sufficient to reproduce results ^[54]. These controls strengthen evidence, but a buyer still needs to reproduce it on the intended platform.

Vendor materials can help locate differences. AMD provides step-by-step reproduction instructions for its measurements ^[55]. NVIDIA labels some later figures as post-submission and not yet verified by MLCommons ^[56]. Such labeling should flow into the evidence record and prevent an unofficial figure from entering an audited ranking.

Convert the shortlist into a pilot

• Clone by commit: save the final v6.1 repository commit and exact result paths.

• Freeze inputs: record model weights, tokenizer, dataset, precision, calibration, and accuracy target.

• Freeze hardware: record part numbers, firmware, CPU, memory, storage, NICs, switches, cooling, and topology.

• Freeze software: record OS, kernel, drivers, runtime, framework, server, compiler, containers, and orchestration.

• Run accuracy first: reject a configuration that misses the required benchmark or application quality threshold.

• Run benchmark performance: reproduce the applicable scenario and retain full logs.

• Replay the buyer trace: use real prompt lengths, output lengths, arrival distribution, concurrency, model mix, and retrieval corpus.

• Measure service behavior: capture throughput, TTFT, TPOT, tail latency, errors, recovery, and sustained utilization.

• Measure facilities: meter node or rack input power and validate cooling behavior at steady load.

• Apply acceptance thresholds: use buyer-supplied limits for quality, latency, availability, power, acoustics, and operations.

• Reconcile economics: attach dated hardware, network, support, installation, colocation, power, and staffing quotes.

• Archive evidence: preserve manifests, commands, logs, telemetry, deviations, and signed acceptance results.

NIST warns that common software performance practices can yield results that are plausible but wrong ^[57]. SPEC's current run rules likewise state that energy metrics must not be estimated ^[58]. The pilot is therefore an evidence-transfer exercise, not a ceremonial rerun.

Data Analysis and Evidence

Table 2 shows how a few exact repository rows should be recorded. It is an evidence sample, not a cross-vendor leaderboard.

The 4x to 16x Qwen3-VL pair has a scale factor of four. Its descriptive efficiency is 230.088792881784 / (58.395875238853066 × 4) = 0.9850, or 98.50%. This supports a narrow statement: the published larger configuration delivered 98.50% of ideal linear scaling relative to that smaller published configuration. It does not isolate networking as the cause of the 1.50% gap, prove production scaling, or authorize comparison with the Interactive rows.

The eight-GPU and 512-GPU MI355X quotients show why per-accelerator output should not become a ranking by itself. Their topology and replica strategies differ, and their results should not be compressed into a universal summary score.

Table 3 turns benchmark output into a buyer-owned financial and facility model. Blank fields are deliberate.

MLPerf power policy includes all components sensitized by LoadGen ^[66] and requires power and performance from the same run ^[67]. The selected v6.1 rows have no published power result, so this article does not calculate energy per token. That absence is a pilot requirement, not an invitation to substitute TDP.

Total cost should also extend beyond purchase price. U.S. Department of Energy guidance tells buyers to look beyond initial cost to total cost of ownership ^[68]. Berkeley Lab's projected 325 to 580 TWh range for U.S. data centers in 2028 illustrates the uncertainty created by equipment, utilization, and cooling assumptions ^[69]. The report itself notes that limited data availability constrains analysis ^[70]. NVIDIA also cautions that an illustrative average rack heat load cannot replace precise capacity planning ^[71]. Facility inputs should therefore retain ranges.

Implications and Future Directions

The transition toward endpoints, RAG, and agentic work makes system-level benchmarking more useful, but also increases dependence on pipeline architecture. Intel says its RAG submission divided work between CPU and GPUs ^[72]. NVIDIA says its preview Vera Rubin entries used disaggregated serving with separate prefill and decode plus expert parallelism ^[73]. CoreWeave explicitly labels a reported per-GPU figure as not verified by MLCommons ^[74], showing why verification status belongs in the evidence record. Future procurement will increasingly evaluate orchestration and networking alongside silicon.

Three practices follow:

• Preserve raw evidence. Save result paths, system descriptions, configs, logs, code, and commit identifiers in the RFP file.

• Separate eligibility from scoring. Availability, support, security, facility fit, and reproducibility are gates. Performance and economics are scored only after those gates pass.

• Demand dated buyer evidence. Quotes, lead times, warranty, support, metered power, and installation constraints belong beside the benchmark row.

• Use preview for planning only. A preview result can justify technical evaluation, not a delivery assertion or booked depreciation schedule.

• Keep Open as a laboratory. Techniques from Open may improve a later pilot, but should not enter a Closed price-performance ranking unchanged.

MLPerf's process provides meaningful safeguards: each submitter must review at least one other submission ^[75], and a review committee oversees publication ^[76]. Those controls make v6.1 a strong shortlist input. They do not replace commercial diligence or application validation.

Frequently Asked Questions (FAQs)

How should buyers interpret MLPerf Inference 6.1 results?

Start with the workload, then filter to identical suite, division, benchmark, accuracy target, scenario, availability class, unit, and constraints. Read the system and configuration files before calculating ratios. Treat the result as evidence for selecting a pilot, not proof of an application SLO.

How can GPUs be compared using MLPerf?

Compare complete systems in a valid cohort. GPU labels alone omit host CPUs, memory, interconnect, node count, cooling, runtime, quantization, and inference software. When accelerator counts are verified, divide system output by count only as a labeled descriptive normalization. Do not predict a single-device result from it.

What is the difference between Server and Offline results?

Offline sends all samples at once. Server evaluates query rates under random arrivals and latency obligations. Aggregate Offline throughput answers a batch-capacity question; Server throughput answers a constrained request-serving question. Neither substitutes for Interactive TTFT and TPOT.

Can MLPerf show performance per watt?

Only when a valid power result exists for the exact run. Power policy requires full-system measurement and same-run performance. For the sampled v6.1 rows, no power result is published. A GPU's board-power specification is not measured system energy.

Does MLPerf provide GPU price-performance or TCO?

No. Add a dated quote with inclusions and exclusions to a locally validated result relevant to the SLO. Model support, networking, storage, installation, colocation, power, cooling, staffing, utilization, financing, and risk separately. Never insert an estimated street price into an audited comparison.

Which v6.1 submissions deserve an RFP shortlist?

Rows deserve a shortlist when their workload matches, cohort is valid, system is commercially eligible for the procurement date, artifacts are complete, facility requirements fit, and the supplier can support a reproduction pilot. A Preview entry may enter a technology watchlist, but not an Available-only delivery ranking.

What are MLPerf's limitations for GPU buyers?

MLPerf measures declared systems on standardized workloads. It does not supply the buyer's price, lead time, support scope, facility cost, production utilization, application quality, or traffic distribution. Its public result can establish a reproducible technical baseline, but it cannot establish total cost of ownership or prove a private inference SLO without local evidence.

What are the main red flags in vendor proposals?

• Cherry-picked scenario: a percentage combines Offline, Server, or Interactive without disclosure.

• Mixed divisions: Open is ranked against Closed as if the model and quality conditions were identical.

• Preview sold as delivery: benchmark availability status is treated as a promised ship date.

• Aggregate record without scale: accelerator count, nodes, fabric, or cooling is omitted.

• Per-GPU arithmetic across unlike systems: topology and replica strategy are ignored.

• No result path: a percentage cannot be traced to a summary row and submission artifacts.

• Unverified update: a post-submission result is presented as MLCommons-verified.

• TDP energy claim: board power replaces metered full-system energy.

• Invented economics: price, utilization, tariff, or duty cycle lacks dated buyer evidence.

• Benchmark equals SLA: passing public accuracy and latency constraints is treated as proof for a different production trace.

Conclusion

MLPerf Inference 6.1 is a rich procurement evidence set, but its value depends on disciplined reading. The release adds RAG, agentic inference, a more production-like API harness, new optimizations, and new available and preview hardware. Those additions broaden coverage while making cohort construction more important.

The defensible workflow begins with workload fit, filters like with like, reads the complete system and software configuration, and labels every normalization as descriptive. The clean four-to-16 accelerator example shows how scale efficiency can be calculated without pretending it predicts another deployment. The lack of power results for the sampled rows shows equal discipline: leave energy blank until it is measured.

For an enterprise buyer, the public benchmark should end where internal evidence begins. Freeze the exact artifacts, reproduce accuracy and performance, replay the intended traffic, meter power, validate facility constraints, and attach dated commercial quotes. Only then can benchmark output become an RFP score, a price-performance ratio, or an investment case.