SXM vs PCIe GPUs: Form Factor, Cooling and Use Cases

SXM vs PCIe GPUs: Form Factor, Cooling and Use Cases

Executive Summary

SXM (Server PCI Express Module) and PCIe (Peripheral Component Interconnect Express) are the two form factors NVIDIA uses to package its data center graphics processing units (GPUs), and the choice between them shapes bandwidth, power, cooling, and total cost of ownership more than the underlying silicon does. NVIDIA sells H100 versions in both packages, but they are not equivalent single-GPU configurations: H100 SXM5 has 132 streaming multiprocessors (SMs), 16,896 FP32 CUDA cores, and 80GB of HBM3 memory, while H100 PCIe has 114 SMs, 14,592 FP32 CUDA cores, and 80GB of HBM2e memory. The SXM version ships on a proprietary mezzanine module that plugs into an HGX baseboard and draws up to 700 watts (W) of thermal design power (TDP), while the PCIe version is a standard dual-slot, air-cooled card ^[1]. The interconnect gap is an important variable for multi-GPU workloads: SXM GPUs communicate over NVLink at up to 900 gigabytes per second (GB/s) of bidirectional bandwidth per GPU on Hopper-generation hardware, while a PCIe Gen5 x16 link provides 64GB/s in each direction, or 128GB/s aggregate bidirectional bandwidth. PCI-SIG's 128GB/s figure applies to a 32-lane PCIe 5.0 link, not x16. ^{[2] [3]}. That bandwidth advantage is why every major hyperscale cloud, including Amazon Web Services (AWS) EC2 P5, Google Cloud A3, and Microsoft Azure ND H100 v5, provisions its flagship multi-GPU training instances exclusively on SXM hardware wired through NVIDIA's NVSwitch fabric ^{[4] [5]}.

Pricing data gathered directly from provider pages as of July 2026 confirms a consistent, if modest, premium for SXM within the same provider: Lambda charges $3.29 per GPU-hour for H100 PCIe versus $4.29 for H100 SXM, and CoreWeave charges $4.25 versus $4.76 for the equivalent pair ^{[6] [7]}. AWS lists its 8-GPU, SXM-only p5.48xlarge Capacity Block at an effective $5.191 per H100-hour in US East (N. Virginia), US East (Ohio), US West (Oregon), and US West (N. California); Capacity Blocks are reservations, not on-demand instances. Google Cloud's a3-highgpu-8g instance works out to roughly $11.06 per H100 per hour on demand ( AWS Capacity Blocks pricing ^[8]. On the power and cooling side, Uptime Institute's most recent global survey found that average server rack densities remain below 6 kilowatts (kW), with most operators reporting no racks beyond 20kW, even as 700W-class SXM accelerators push individual 8-GPU nodes well past that envelope ^[9].

The practical answer to "SXM versus PCIe GPU" depends on whether a workload is bound by inter-GPU communication. Large-scale distributed training of large language models (LLMs), the kind Meta ran across 24,576 NVIDIA Tensor Core H100 GPUs per cluster to train Llama 3, and the kind NVIDIA used to set MLPerf training records on 3,584 H100 GPUs with CoreWeave, requires SXM's NVLink and NVSwitch fabric because gradient synchronization across GPUs becomes the bottleneck at scale ^[10]. Conversely, single-GPU inference, fine-tuning workloads that fit on one or two accelerators, and cost-sensitive deployments favor PCIe cards, which cost less to buy, fit standard air-cooled chassis, and do not require a specialized HGX baseboard. This report walks through the architecture, cooling requirements, benchmark evidence, cloud pricing, and five named deployments that illustrate when each form factor makes financial and technical sense, closing with a decision framework organizations can apply directly to procurement.

Five real-world cases anchor the analysis: Meta's 24,576-GPU Grand Teton clusters, NVIDIA's own Eos supercomputer built from 4,608 SXM-based H100 GPUs, the Department of Energy's NERSC Perlmutter system (whose A100 SXM4 GPUs use PCIe for host connectivity and NVLink for GPU-to-GPU communication), AWS's single-GPU p5.4xlarge instance used by insurer AON for actuarial modeling, and CoreWeave's record-setting MLPerf training submissions on both HGX H100 and the newer GB300 NVL72 ^{[11] [12]}. Taken together, the evidence points to a durable rule of thumb rather than a universal winner: SXM can be advantageous for tightly coupled multi-GPU workloads, while PCIe can be the more economical, operationally simpler choice when the workload, server design, and communication profile do not justify an HGX-scale system.

Introduction and Background

Several NVIDIA accelerator families, including A100 and H100, have been offered in both SXM and PCIe form factors, but availability varies by product. PCIe is a decades-old, universal expansion standard that PCI-SIG describes as having "served as the de facto interconnect of choice for nearly two decades," and the same slot design that carries a network card, a storage controller, or a gaming graphics card also accepts a data center GPU shaped as a dual-slot card ^[13]. SXM, by contrast, is NVIDIA's own high-performance mezzanine socket, a bare circuit module without an enclosure that bolts face-down onto a specialized system board called HGX, and NVIDIA's HGX platform page describes it as bringing "together the full power of NVIDIA GPUs" with NVLink and networking into a single certified design that server manufacturers build around ^[14]. NVIDIA's own product literature does not publicly spell out what the "SXM" initialism stands for; trade publications that cover the hardware most closely describe it as Server PCI Express Module, and this report adopts that convention while noting the ambiguity ^[15].

The distinction matters because AI training and high-performance computing (HPC) workloads increasingly split a single job across many GPUs simultaneously, and how fast those GPUs can exchange intermediate results, gradients, or activations directly determines whether the cluster spends its time computing or waiting on data transfer. NVIDIA's H100, announced in 2022 and still widely deployed as of July 2026, is available in both packages and uses the Hopper architecture and TSMC 4N process, but the products are not simply mounted differently: H100 SXM5 has 132 SMs and HBM3, while H100 PCIe has 114 SMs and HBM2e ^[16]. The same pattern holds for the older A100 (Ampere generation), available in both SXM4 and PCIe form factors, and the newer H200, which pairs Hopper compute with larger high-bandwidth memory (HBM) ^[17]. It is worth flagging one small but genuine discrepancy in the public record: NVIDIA's original 2022 H100 datasheet listed PCIe TDP as "300-350W (configurable)," while NVIDIA's current live product page lists the same card at "350-400W (configurable)," a revision this report attributes to a later specification update rather than an error, since the SXM figure of "Up to 700W (configurable)" has remained consistent across both versions ^{[18] [1]}.

This report answers the question "SXM vs PCIe GPU" comprehensively: what each form factor is, how their interconnects differ, what cooling and power infrastructure each requires, how they perform on published benchmarks, what they cost to rent from major clouds, and which real-world deployments illustrate the tradeoffs. It also addresses the related queries practitioners search for, including what an SXM GPU is, how SXM and PCIe compare specifically for the H100 and A100, SXM cooling requirements, NVLink behavior across the two form factors, and concrete guidance for when to choose one over the other. All figures are anchored to publicly available vendor documentation, cloud pricing pages, standards-body specifications, and named deployment case studies gathered and verified as of July 2026; where sources disagree or data has changed across product generations, this report flags the discrepancy rather than presenting a false consensus.

SXM GPUs: Architecture and Interconnect

Capabilities

An SXM GPU is a bare module, exposing its die and memory stack for direct contact cooling, that mounts onto NVIDIA's proprietary HGX baseboard rather than plugging into a conventional PCIe slot ^[19]. In the eight-GPU HGX H100 design, four shared third-generation NVSwitch chips connect all eight GPU positions. This gives GPU pairs a single-hop path without routing traffic through the host central processing unit (CPU); AWS describes this design as letting "each GPU communicate with every other GPU in the same instance with single-hop latency" ^[20]. An H100 has 18 NVLink links that provide up to 900GB/s of aggregate bidirectional NVLink bandwidth per GPU; NVSwitch enables concurrent GPU-pair communication, while that traffic shares each GPU's aggregate NVLink bandwidth, and NVIDIA's datasheet describes the design as delivering "over 7X the bandwidth of PCIe Gen5" for GPU-to-GPU traffic, while also stating the NVLink Switch System "delivers 9X higher bandwidth than InfiniBand HDR on the NVIDIA Ampere architecture" for scale-out clusters of up to 256 H100 GPUs ^[21]. Beyond the NVLink fabric itself, NVIDIA describes its broader HGX networking stack as "combining NVIDIA NVLink scale-up, NVIDIA Quantum InfiniBand and Spectrum-X Ethernet scale-out, Spectrum-XGS Ethernet multi-data-center scale-across" into a single full-stack fabric purpose-built for GPU clusters, a layered design PCIe-only servers do not replicate ^[22]. Because the socket is not power-constrained by a shared motherboard bus, SXM modules also support materially higher TDP than their PCIe siblings, and that extra power headroom is what lets NVIDIA clock SXM parts higher and pair them with faster memory: H100 SXM ships with 80 gigabytes (GB) of HBM3 at 3.35 terabytes per second (TB/s) of bandwidth, versus the original H100 PCIe's HBM2e running at roughly 2TB/s ^[23].

HGX boards hold four or eight GPUs and form the core of NVIDIA's own DGX systems as well as third-party servers certified under the NVIDIA-Certified Systems program. NVLink bandwidth has scaled with every architecture generation since: fourth-generation NVLink on Hopper delivers 900GB/s per GPU, fifth-generation NVLink on Blackwell (used in the B200 and the rack-scale GB200 NVL72) delivers 1,800GB/s, and NVIDIA's sixth-generation NVLink, introduced with the Rubin platform, targets 3,600GB/s of bandwidth per GPU, more than fourteen times a PCIe Gen6 link ^[24]. NVIDIA's newest HGX Rubin NVL8 board pushes this further still, integrating "eight NVIDIA Rubin GPUs with sixth-generation high-speed NVLink interconnects, delivering up to 10x more token factory throughput versus HGX B200" ^[25].

SXM also supports NVIDIA's Multi-Instance GPU (MIG) partitioning technology, which lets administrators split a single physical GPU into as many as seven fully isolated instances, each with dedicated memory, cache, and compute cores; NVIDIA says this "gives administrators the ability to support every workload, from the smallest to the largest, with guaranteed quality of service (QoS)," and separately notes the feature can unlock "up to 7x more GPU resources on a single GPU" for teams that would otherwise need seven separate cards ^{[26] [27]}. NVIDIA notes that without MIG, "different jobs running on the same GPU...compete for the same resources," while MIG guarantees quality of service for each tenant ^[28]. MIG is available on both SXM and PCIe GPUs across "NVIDIA Rubin, NVIDIA Blackwell, and NVIDIA Hopper GPUs," so partitioning flexibility is not by itself a reason to prefer one form factor over the other ^[29]. On the older A100 generation, NVIDIA describes the same capability more simply: "An A100 GPU can be partitioned into as many as seven GPU instances, fully isolated at the hardware level with their own high-bandwidth memory, cache, and compute cores," a feature available identically on A100 SXM and A100 PCIe cards ^[30].

Adoption

SXM is the default configuration for every hyperscale cloud's flagship AI training instance as of mid-2026. AWS's EC2 P5 family provides up to eight H100 GPUs per instance with up to 900GB/s of NVSwitch interconnect, for a total of 3.6TB/s of bisectional bandwidth across the instance, and up to 3,200 gigabits per second (Gbps) of cross-node networking through its second-generation Elastic Fabric Adapter (EFA) ^[31]. Google Cloud's A3 supercomputer instances pair eight H100 SXM GPUs with 3.6TB/s of bisectional bandwidth through NVSwitch and NVLink 4.0, plus custom 200Gbps networking chips that Google says provide "up to 10x more network bandwidth compared to our A2 VMs," its prior A100 generation ^[32]. Microsoft's Azure ND H100 v5 series gives each of its eight GPUs a dedicated, topology-agnostic 400Gbps NVIDIA Quantum-2 InfiniBand connection alongside NVLink 4.0 for intra-VM GPU communication, and pairs that with "96 physical fourth Gen Intel Xeon Scalable processor cores" per virtual machine so the host CPU does not become a secondary bottleneck; Microsoft adds that these instances "provide excellent performance for many AI, ML, and analytics tools that support GPU acceleration" such as "TensorFlow, Pytorch, Caffe, RAPIDS, and other frameworks" ^{[33] [34]}. Microsoft states that "ND H100 v5-based deployments can scale up to thousands of GPUs with 3.2 Tbps of interconnect bandwidth per VM" ^[35]. NVIDIA's own announcement of the Azure instance highlights that H100's "latest NVLink technology...lets GPUs talk to each other at 900GB/sec," and adds that "the inclusion of NVIDIA Quantum-2 CX7 InfiniBand with 3,200 Gbps cross-node bandwidth ensures seamless performance across the GPUs at massive scale, matching the capabilities of top-performing supercomputers globally," crediting the combination with helping the ND H100 v5 VMs "achieve up to 2x speedup in LLMs like the BLOOM 175B model for inference versus previous generation instances" ^{[36] [37] [38]}.

Adoption is not limited to public cloud. Meta built its 24,576-GPU Llama 3 training clusters on its own Grand Teton SXM platform, an open GPU hardware design it contributed to the Open Compute Project, choosing the form factor specifically because it "integrate[s] power, control, compute, and fabric interfaces into a single chassis for better overall performance, signal integrity, and thermal performance" ^[39]. NVIDIA's own DGX SuperPOD reference architecture, which underpins its Eos research supercomputer, is likewise built entirely from eight-GPU SXM DGX H100 nodes, each including "two NVIDIA BlueField-3 DPUs" for networking, storage, and security offload plus "Eight NVIDIA ConnectX-7" Quantum-2 InfiniBand networking adapters providing 400Gbps of throughput per adapter, connected by "a fourth-generation NVLink, combined with NVSwitch, [that] provides 900 gigabytes per second connectivity between every GPU in each DGX H100 system" ^{[40] [41] [42]}.

Strengths and Limitations

SXM's central strength is bandwidth density: NVSwitch gives every GPU on a board a full-mesh, single-hop connection to every other GPU, which keeps multi-GPU training compute-bound rather than communication-bound as job sizes grow. Its second strength is thermal and power headroom, since the HGX baseboard delivers dedicated high-wattage power to each GPU position rather than sharing a general-purpose motherboard bus, making it practical to sustain 700W-class parts with a cold plate mounted directly on the exposed die ^[43].

The limitations are structural. SXM modules require a purpose-built HGX baseboard rather than a general-purpose motherboard, which most organizations buy pre-integrated inside an HGX-certified or DGX chassis rather than assembling themselves. NVIDIA's HGX platform page describes HGX baseboards with eight Rubin, Blackwell, or Blackwell Ultra SXM modules. That describes the HGX platform configuration; it does not establish that every GPU product based on those architectures lacks a PCIe option ^[44]. That baseboard, plus the higher per-GPU power draw, generally pushes SXM systems toward liquid cooling once density climbs, which raises facility requirements relative to a conventional air-cooled rack, a tradeoff examined in more detail in the Data Analysis section below. SXM configurations are also typically sold and priced as fixed four- or eight-GPU units rather than the single-card increments PCIe supports, which reduces flexibility for organizations that only need one or two accelerators.

PCIe GPUs: Architecture and Deployment Model

Capabilities

A PCIe data center GPU is a standard dual-slot, air-cooled card that slots into the same x16 expansion connector used by consumer graphics cards, network interface cards, and storage controllers, requiring no proprietary baseboard ^[45]. The PCI Special Interest Group (PCI-SIG), which governs the standard, has doubled per-lane bandwidth with each generation: PCIe 1.0 provided 2.5 gigabits per second per lane per direction, PCIe 2.0 raised that to 5.0Gb/s, PCIe 3.0 to 8.0Gb/s, and "PCIe 4.0 added support for 16.0 Gigabits/second/Lane/direction of raw bandwidth," with PCIe 5.0 reaching 32.0 gigatransfers per second (GT/s), which across a 16-lane link provides 64.0GB/s in each direction, or 128.0GB/s aggregate bidirectional bandwidth ^{[46] [3]}. The newer PCIe 6.0 specification doubles bandwidth again, to "64.0 GT/s raw data rate and up to 256.0 GB/s via x16 configuration" using Pulse Amplitude Modulation with four levels (PAM4) signaling, with "Lightweight Forward Error Correct (FEC) and Cyclic Redundancy Check (CRC)" added specifically to "mitigate the bit error rate increase associated with PAM4 signaling," and PCI-SIG notes the new specification "maintains backwards compatibility with all previous generations of PCIe technology." A PCIe 6.0 device can therefore operate in an older compatible host, but the link runs at the highest speed supported by that configuration; realizing PCIe 6.0 bandwidth requires a compatible host/root complex and platform infrastructure. As of July 2026, most deployed H100 and A100 PCIe cards still use Gen5 or Gen4 connections ^{[47] [48] [49]}. That ceiling, which applies to both a GPU's connection back to the host CPU and, absent a bridge, between two PCIe GPUs in the same server, is the fundamental constraint distinguishing PCIe from SXM's NVLink fabric.

Because PCIe GPUs share a conventional motherboard's power delivery, their TDP is capped well below SXM's. To recover some multi-GPU bandwidth without a full HGX baseboard, NVIDIA sells NVLink Bridge kits that connect up to two PCIe GPUs directly: on the A100 generation, NVIDIA states that "NVLink is available in A100 SXM GPUs via HGX A100 server boards and in PCIe GPUs via an NVLink Bridge for up to 2 GPUs," delivering 600GB/s between the pair ^[50]. A comparable bridged configuration exists for the specialized H100 NVL card, well short of the eight-way, full-mesh 900GB/s fabric that NVSwitch provides but far faster than routing through the PCIe bus and host CPU. NVLink-bridged PCIe availability is product-specific and should be verified against the relevant GPU's specifications. It should not be inferred from the availability of HGX SXM systems, because NVIDIA also sells Blackwell-based RTX PRO 6000 Server Edition GPUs in a PCIe Gen5 x16, dual-slot form factor.

Adoption

PCIe GPUs remain the default choice for single- and dual-GPU deployments across independent GPU clouds and on-premises servers. Lambda sells H100 PCIe on-demand at $3.29 per GPU-hour, alongside A100 PCIe at $1.99 per GPU-hour in its smaller instance tiers, while also offering multi-GPU 1-Click Clusters that "spin up 1-Click Clusters featuring 16-2,000+ interconnected HGX B200 and H100 GPUs for large-scale AI workloads" for customers that outgrow single cards ^{[51] [52]}. CoreWeave lists NVIDIA H100 PCIe at $4.25 per hour separately from its NVLink-connected A100 and HGX H100 offerings on its classic pricing page, and interestingly prices A100 80GB PCIe and A100 80GB NVLink identically at $2.21 per hour, showing that pricing premiums for SXM are not universal across every GPU generation or provider; the same catalog also itemizes vCPUs and memory separately from the GPU line item, billed by the hour per core and per gigabyte, a granular model independent GPU clouds commonly use instead of the bundled instance pricing hyperscalers favor ^[53]. PCIe form factors are also favored in HPC clusters that pair a modest number of GPUs with a CPU-heavy node design: the National Energy Research Scientific Computing Center's (NERSC) Perlmutter supercomputer, for example, equips each GPU node with four NVIDIA A100 GPUs connected to the host over a "PCIe 4.0 GPU-CPU connection," even though the GPUs themselves communicate with each other over "4 third generation NVLink connections between each pair of GPUs," illustrating that PCIe and NVLink are not mutually exclusive within a single node design ^{[11] [54]}.

Cloud providers also offer single-GPU instances for workloads that do not need multi-GPU scaling, a pattern illustrated by AWS's smallest P5 configuration and detailed in the AWS EC2 P5 and AON case study below. This example establishes that a single-GPU workload does not need GPU-to-GPU interconnect; it does not identify the P5 GPU as a PCIe-form-factor deployment.

Strengths and Limitations

PCIe's core strength is compatibility and cost. Because it uses the same slot as every other server expansion card, a PCIe GPU can go into an existing, general-purpose chassis without a proprietary baseboard, keeping capital costs and supply chain flexibility higher than an SXM deployment requires ^[55]. Lower TDP per GPU also means PCIe systems more often stay within the reach of conventional air cooling, avoiding the facility upgrades that high-density liquid-cooled racks require.

The limitation is the same bandwidth constraint described above: a PCIe Gen5 x16 link provides 64GB/s in each direction, or 128GB/s aggregate bidirectional bandwidth, and even a bridged pair does not provide the eight-way NVSwitch topology available in an HGX system. The practical impact depends on the workload's communication pattern, as independent MLPerf submissions illustrate. Chip maker Qualcomm's MLPerf Inference results noted that its CloudAI100 accelerators, "each limited to 75W TDP," beat "eight Nvidia H100 (PCIe)" on throughput per watt for image classification and object detection, while NVIDIA's SXM-based DGX-H100 system topped the same round's language-model benchmarks, underscoring that the PCIe H100 configuration used in that comparison was not NVIDIA's highest-throughput entry ^[56]. PCIe deployments also generally require purchasing an NVLink Bridge separately, and only in pairs, to recover any of the multi-GPU bandwidth SXM provides natively across all eight positions on an HGX board ^[50].

Feature Comparison

Table 1 below consolidates the official NVIDIA specifications for the H100 and A100 generations across both form factors, drawing on NVIDIA's datasheets. Figures reflect the vendor's published configurable maximums as of July 2026 and should be read as ceilings rather than typical sustained values.

Sources: ^{[16] [57] [58]}

The table makes two points clear. First, form factor can coincide with material product-configuration differences: for H100, SXM5 has more enabled SMs and FP32 CUDA cores, HBM3 rather than HBM2e, and higher memory bandwidth than H100 PCIe. Power delivery and interconnect are therefore important differences, but not the only ones. Second, the interconnect gap compounds with GPU count: on an 8-GPU HGX H100 board, the four shared NVSwitches provide single-hop connectivity between GPU pairs, and each H100 contributes up to 900GB/s of aggregate bidirectional NVLink bandwidth. Concurrent traffic shares that per-GPU aggregate bandwidth. Eight PCIe cards without bridges instead use the system's PCIe topology for peer traffic, and NVLink Bridge configurations are product-specific and do not form the eight-GPU NVSwitch fabric. It is also worth noting that NVIDIA's rival AMD ships an analogous but non-interchangeable mezzanine format for its Instinct accelerators: the MI300X uses the Open Compute Project's OCP Accelerator Module (OAM) format, itself connected to the host over a "PCIe 5.0 x16" bus but relying on AMD's own Infinity Fabric links, eight of them, rated at up to "128 GB/s" of peak per-link bandwidth for GPU-to-GPU traffic, on a card whose own performance footnotes describe "192 GB HBM3 memory capacity and 5.325 TFLOPS peak theoretical memory bandwidth" ^{[59] [60] [61]}. This confirms the SXM-versus-PCIe tradeoff described here is an industry pattern rather than an NVIDIA-only design choice.

Performance and Benchmarks

Independent and vendor-reported benchmarks show that multi-GPU scaling can be strongly affected by interconnect topology. For a single-GPU workload, GPU-to-GPU interconnect is irrelevant, but performance can still differ materially between the exact SXM and PCIe products because their enabled compute units, memory type, memory bandwidth, and power limits can differ. In NVIDIA's own MLPerf Training v3.0 submissions, the company's SXM-based H100 platform set new per-accelerator performance records across every workload category and delivered up to 3.1 times more performance per accelerator than the prior A100 generation's MLPerf Training v2.1 results ^[62]. At scale, NVIDIA and CoreWeave's joint submission trained a 175-billion-parameter GPT-3 model to convergence in just 10.9 minutes using 3,584 H100 SXM GPUs on CoreWeave's HGX H100 infrastructure, a result the companies describe as demonstrating "89% performance scaling efficiency even when moving from hundreds to thousands of H100 GPUs" ^[63]. NVIDIA's smaller-scale submissions reinforce the same scaling story: a 512-GPU H100 run trained the same GPT-3 benchmark in 64.3 minutes, and "scaling to 768 GPUs on the same Pre-Eos system reduced time-to-train to 44.8 minutes for near-linear scaling efficiency" ^[64]. That kind of scaling across thousands of GPUs is only achievable because NVLink and NVSwitch keep GPU-to-GPU communication off the comparatively slow PCIe bus.

Inference benchmarks tell a more nuanced story. MLPerf Inference results reported by EE Times show NVIDIA's SXM-based DGX-H100 system "topping the table for Bert-Large results" in most scenarios, with one exception where a Dell PowerEdge server using eight H100s (with a different host CPU) edged ahead in a single offline-mode category, underscoring that host CPU and system integration, not just the GPU form factor, affect inference throughput ^[65]. The same report found that scaling from one to eight GPUs within a DGX-H100 system stayed close to a perfect 8.0 times multiplier for most workloads, with only modest outliers in specific scenarios, again reflecting NVSwitch's near-linear scaling characteristics ^[66]. At the same time, third-party inference accelerators demonstrated that PCIe-connected H100 systems are not automatically the most power-efficient option for every workload: Qualcomm's CloudAI100 chips outperformed eight PCIe H100 GPUs on queries per second per watt for image classification and object detection by a factor of 1.5 to 2.1 times, a reminder that raw interconnect bandwidth is only one input to inference efficiency ^[67]. On the cloud side, Google reports its H100-based A3 instances deliver "up to a 30x inference performance boost when compared to our A2 VM's that are powered by NVIDIA A100 Tensor Core GPU," a comparison that mixes generational GPU improvements with the interconnect upgrade from A2 to A3, and separately credits the platform's scale for "up to 26 exaFlops of AI performance," a figure that depends entirely on stitching thousands of NVSwitch-connected GPUs together rather than any single accelerator's raw throughput ^{[68] [69]}.

Generational comparisons reinforce the pattern. NVIDIA's H200 datasheet, which specifies both an SXM and a PCIe-form-factor "NVL" variant, reports that H200 doubles inference performance relative to H100 on large models such as Llama 2 70B, and delivers up to 110 times faster time-to-results on select HPC applications when measured on the SXM configuration against an H100 SXM baseline, while stating that H200's added performance arrives "all within the same power profile as the H100 Tensor Core GPU" ^{[17] [70]}. Because both H200 variants share the same 141GB of HBM3e memory and 4.8TB/s of memory bandwidth, and differ mainly in TDP (up to 700W for SXM versus up to 600W for the PCIe-based NVL card) and interconnect topology, the H200 comparison illustrates that even as memory capacity converges across form factors, interconnect remains the primary performance differentiator at scale.

Data Analysis and Evidence

Cloud pricing is one quantitative input to infrastructure decisions. Table 2 below compiles published effective per-GPU-hour rates gathered from provider pricing pages in July 2026; these rates are not all offered under the same purchasing model.

Sources: ^{[71] [72] [73] [8]}

AWS rows are Capacity Block reservation rates, paid as a reservation fee and expressed by AWS as effective hourly rates; they are not on-demand rates.

Table 2 shows a consistent pattern across the two providers that sell both form factors side by side: Lambda charges roughly 30 percent more for H100 SXM than H100 PCIe ($4.29 versus $3.29), and CoreWeave charges roughly 12 percent more ($4.76 versus $4.25), which is a modest premium relative to the sevenfold interconnect bandwidth difference the hardware provides. It also shows that the largest hyperscalers, AWS and Google Cloud, do not offer PCIe-based multi-GPU instances at flagship tiers at all; their high-end training products are SXM-only, which is consistent with the architecture and adoption evidence discussed above. Google Cloud's A3 pricing, at roughly $11.06 per H100 per hour, sits notably above AWS and the independent clouds, a discrepancy this report flags rather than resolves, since provider pricing reflects networking, support, and regional cost structures beyond the GPU itself. The table should not be read as evidence that NVIDIA has no PCIe Blackwell GPU products. AWS publishes Capacity Block effective hourly reservation rates for its p6 and GB200 UltraServer offerings, but those reserved rates should not be compared directly with on-demand rates or used to infer an SXM-versus-PCIe premium ^[73]. AWS's Capacity Block pricing model itself is worth noting as a structural difference from ordinary on-demand billing: it requires reserving GPU capacity for a fixed future time window, and AWS explains that "when you use an EC2 Capacity Block, you pay for the operating system you use when your instances are running," a mechanism the company introduced specifically to guarantee availability for large, time-boxed training runs on scarce SXM-based instances ^[74]. Google Cloud's own accelerator pricing page similarly confirms its lowest-cost A100 tier, the single-GPU A2 Standard configuration, remains an SXM part rather than a PCIe one, underscoring how thoroughly SXM has displaced PCIe across every hyperscaler's catalog regardless of instance size ^[75].

Power and cooling economics are the other major quantitative dimension. Uptime Institute's most recent global data center survey, its 13th annual edition, found that "server rack densities are climbing steadily, but slowly," with average densities remaining below 6kW per rack, and the survey separately notes that "most operators do not have any racks beyond 20 kW," a finding whose authors say "suggests the widespread use of direct liquid cooling is not imminent" across the industry as a whole ^{[76] [77]}. That baseline is a meaningful shift from just a few years earlier: an Uptime Institute survey of the industry found "the mean average density in our 2020 survey sample was 8.4 kW/rack," with "the most common density" then sitting at "5-9 kW/rack," and separately reported that "racks with densities of 20 kW and higher are becoming a reality for many data centers," even though such high densities remained concentrated among a small number of hyperscale operators rather than the industry at large ^{[78] [79] [80]}. That baseline is important context for SXM deployments: an 8-GPU HGX H100 board alone can draw up to 5.6kW from the GPUs before accounting for host CPUs, memory, storage, and networking, meaning a fully populated SXM rack can exceed typical facility provisioning even before reaching the 20 to 25kW threshold that Uptime Institute's research separately identifies as the point where "direct liquid cooling and precision air cooling becomes more economical and efficient" ^[81]. At the extreme end of the roadmap, NVIDIA describes its GB200 NVL72 as a rack-scale, liquid-cooled design that integrates 72 Blackwell GPUs and 36 Grace CPUs in a single rack, and states that, compared with air-cooled H100 infrastructure, it delivers 25x more performance at the same power while reducing water consumption through that liquid-cooling architecture ^{[82] [83]}. By contrast, the H100 PCIe card remains a conventional air-cooled design, fitting within the density envelope most data centers already support without facility upgrades.

Interconnect standards data supplies the third quantitative pillar. PCI-SIG's own specification confirms the underlying PCIe ceiling that both form factors' host connections share, and describes PCIe 6.0 as "the cost-effective and scalable interconnect solution for data-intensive markets like Data Center, Artificial Intelligence/Machine Learning, HPC," language that positions even the fastest PCIe generation as complementary to, rather than a substitute for, dedicated GPU interconnects like NVLink ^[84]. A 16-lane PCIe 5.0 link, the generation underlying most deployed H100 and A100 PCIe cards, provides 64.0GB/s in each direction, or 128.0GB/s aggregate bidirectional bandwidth; PCIe 6.0 x16 doubles the aggregate figure to as much as 256.0GB/s. These host-link figures remain well below the 900GB/s per-GPU NVLink bandwidth SXM systems achieve today ^[3]. NVIDIA's own generational NVLink table, published alongside its GB200 NVL72 materials, confirms the sixth-generation NVLink figure of 3.6TB/s per GPU cited throughout this report, reinforcing that the interconnect gap versus PCIe widens rather than narrows with each new architecture ^[24].

Case Studies and Real-World Examples

Meta's Grand Teton Clusters: SXM at 24,576-GPU Scale

Meta disclosed in March 2024 that it had built two 24,576-GPU clusters on its in-house Grand Teton SXM platform specifically to train Llama 3, describing the combination of "the efficiency of the high-performance network fabrics within these clusters" and "the 24,576 NVIDIA Tensor Core H100 GPUs in each" as what allowed both cluster versions to support larger and more complex models than its prior AI Research SuperCluster, which had used "16,000 NVIDIA A100 GPUs, in 2022" ^{[85] [86]}. Meta built one cluster on a Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) fabric and the other on NVIDIA Quantum2 InfiniBand specifically to compare scale-out interconnect options, and reported that "both of these solutions interconnect 400 Gbps endpoints," while both clusters shared the same SXM-based Grand Teton compute platform for intra-node GPU communication ^[87]. By the end of 2024, Meta said it planned to grow its total infrastructure to 350,000 NVIDIA H100 GPUs, part of a portfolio delivering compute power equivalent to nearly 600,000 H100s ^[88]. The scale of that build-out is only tractable with SXM's NVSwitch fabric; routing that many GPUs' gradient synchronization traffic over PCIe alone would leave the CPUs and PCIe fabric as the bottleneck long before GPU compute became the limiting factor.

NVIDIA Eos and the DGX SuperPOD: SXM for National-Scale HPC

NVIDIA's own Eos supercomputer, announced alongside the DGX H100 platform, is built from 576 DGX H100 systems for a total of 4,608 H100 GPUs, all connected via SXM modules on HGX baseboards, NVLink Switch System, and Quantum-2 InfiniBand ^[89]. NVIDIA said the resulting DGX SuperPOD architecture delivers a total of "70 terabytes/sec of bandwidth" across the pod, and framed the entire system as designed to "run massive LLM workloads with trillions of parameters" ^[90]. This case illustrates SXM adoption at the vendor's own reference-architecture level: NVIDIA does not offer an equivalent PCIe-based SuperPOD design for large-scale distributed training, reflecting the same bandwidth logic that drove Meta's and the hyperscalers' choices.

NERSC Perlmutter: A Hybrid Deployment That Clarifies the Boundary

The Department of Energy's NERSC facility operates Perlmutter, a system based on the HPE Cray Shasta platform that is "a heterogeneous system comprised of 3,072 CPU-only and 1,792 GPU-accelerated nodes," each pairing a single AMD EPYC CPU with four NVIDIA A100 GPUs ^[91]. Within a node, the four A100 GPUs communicate with each other over four third-generation NVLink connections, each rated at "25 GB/s/direction for each link," a capability the PCIe Adoption section above shows works alongside, not instead of, the node's separate PCIe 4.0 host connection ^[92]. This is a valuable real-world clarification of a common misconception: SXM and PCIe are not mutually exclusive labels for an entire system, since a node can use SXM GPUs with NVLink for intra-node GPU-to-GPU traffic while still relying on PCIe for the separate GPU-to-CPU and GPU-to-network-interface-card paths. Perlmutter's design also demonstrates that four-GPU SXM nodes, rather than the eight-GPU configuration common in hyperscale cloud offerings, remain a common HPC procurement pattern.

AWS EC2 P5 and AON: When a Single-GPU Workload Is Sufficient

Not every production deployment needs multi-GPU interconnect at all. AWS's own EC2 P5 marketing highlights insurer AON's use of the single-GPU p5.4xlarge instance, quoting the company's Global Head of Life Solutions describing how the instance let AON "run machine learning models and economic forecasts that used to take days in just a matter of hours," and noting that "the ability to use a single H100 GPU instance (p5.4xlarge) means we're not only saving time but also optimizing our computational resources" ^{[93] [94]}. AWS's own instance specification confirms that p5.4xlarge does not support GPU peer-to-peer connectivity and states plainly that "GPUDirect RDMA is not supported in P5.4xlarge," the features that matter only once a workload spans multiple GPUs ^[95]. This case underscores a narrower decision principle: for actuarial modeling and similar single-accelerator analytical workloads, multi-GPU interconnect is not a workload requirement. Select a single-GPU instance based on the model's memory needs, published GPU specifications, availability, and price; this example does not establish the economics of a PCIe form factor.

CoreWeave and NVIDIA: SXM at the Frontier of MLPerf Records

CoreWeave's joint MLPerf Training v3.0 submission with NVIDIA used HGX H100 (SXM) infrastructure to train a 175-billion-parameter GPT-3 model at 3,584-GPU scale in 10.9 minutes, a result the companies say demonstrates that "the NVIDIA platform delivers great performance in both on-premises and commercially available cloud instances" ^[96]. More recently, in MLPerf Training v6.0, CoreWeave reported training the DeepSeek-V3 671-billion-parameter mixture-of-experts model in 2.02 minutes on 8,192 NVIDIA GB300 GPUs, calling it "the fastest time recorded in the round across all available-cloud submissions on this benchmark" and noting the run used "the largest GB300 NVL72 cluster ever benchmarked on this workload," a cluster CoreWeave says is "16x the next-largest GB300 submission in the round" ^{[12] [97] [98]}. CoreWeave also reported a 2.8 times wall-clock speedup on its Llama 3.1 405B training benchmark year over year, crediting the combination of "the latest NVIDIA GB300 NVL72 systems with NVIDIA Spectrum-X networking, the topology-aware SUNK scheduler, and CoreWeave Mission Control" ^[99]. NVIDIA documents GB300 NVL72 as a liquid-cooled, rack-scale NVLink architecture with 72 Blackwell Ultra GPUs, nine NVSwitch trays, and direct NVLink connectivity among all GPUs in the rack. That documentation does not identify GB300 as an SXM form factor, so this result is evidence about a rack-scale NVLink system rather than an SXM-versus-PCIe comparison.

Implications and Future Directions

The trajectory of NVIDIA's roadmap suggests that tightly integrated, high-end HGX and rack-scale systems will continue to rely on SXM-class modules, even as the distinction remains less relevant for smaller deployments. This does not mean every Blackwell-based GPU is SXM-only: NVIDIA also sells a Blackwell-based RTX PRO 6000 Server Edition in a PCIe Gen5 x16, dual-slot form factor. That shift pushes frontier-scale training and reasoning workloads further toward the rack-scale, liquid-cooled model that GB200 NVL72 pioneers, where NVIDIA reports a "130 terabytes per second (TB/s) of low-latency GPU communications" domain spanning all 72 GPUs in a single rack, a bandwidth figure with no PCIe analog at any generation ^[100]. NVIDIA also offers a smaller-scale bridged option for organizations that need more than one GPU acting together without committing to a full rack: the "NVIDIA GB200 NVL4...deliver[s] revolutionary performance through a bridge connecting four NVIDIA NVLink Blackwell GPUs unified with two Grace CPUs," compatible with liquid-cooled modular servers rather than the full NVL72 rack ^[101]. Organizations planning multi-year AI infrastructure investments should profile the target workload's communication pattern, GPU count, power envelope, and cooling constraints before selecting an HGX/SXM-class system. Tightly coupled workloads may justify that infrastructure, but GPU count alone does not establish a universal form-factor requirement.

At the same time, PCIe is not disappearing from the inference and edge tiers. MIG partitioning works on both SXM and PCIe-mounted GPUs and, on newer NVIDIA architectures, an administrator "could create two instances with 93GB of memory each, four instances with 46GB each, or seven instances with 23GB each" from a single GB200-class GPU, letting organizations right-size one physical accelerator to many smaller inference workloads regardless of form factor ^[102]. This reduces the practical urgency of choosing SXM purely for utilization reasons. NVIDIA has also continued releasing PCIe-native "NVL" variants for memory-bound inference use cases through the Hopper generation, and the H200 datasheet explicitly frames the PCIe-form-factor H200 NVL as "the ideal choice for customers with space constraints within the data center," targeting large language model serving where a single GPU lacks sufficient memory but an eight-GPU SXM system would be over-provisioned ^[103].

Cooling infrastructure investment is the practical constraint most likely to determine adoption speed. Uptime Institute's finding that most data centers still operate well below the 20 to 25kW threshold where liquid cooling becomes economically favorable implies that a large share of the installed base cannot host SXM-based, multi-GPU systems without capital investment in power distribution and cooling upgrades, even if the compute economics otherwise favor SXM ^[104]. This is one reason independent GPU clouds like Lambda and CoreWeave continue to sell PCIe capacity alongside SXM rather than migrating entirely to liquid-cooled fleets, as reflected in both providers' published rate cards: a meaningful share of demand, particularly fine-tuning, batch inference, and research prototyping, does not need SXM's bandwidth and would rather avoid its facility requirements and price premium ^[105].

The following decision matrix, Table 3, synthesizes the evidence above into a practical procurement guide organized by workload type.

Frequently Asked Questions (FAQs)

What is an SXM GPU? An SXM GPU is an NVIDIA data center graphics processing unit packaged as a bare mezzanine module rather than an enclosed card, designed to mount onto a proprietary HGX baseboard. In the eight-GPU HGX H100 design, four shared NVSwitch chips provide single-hop GPU-to-GPU communication over NVLink; each H100 has up to 900GB/s of aggregate bidirectional NVLink bandwidth. It cannot be installed in a conventional server motherboard slot and instead ships pre-integrated inside an HGX-certified or DGX chassis.

How does SXM compare to PCIe for the H100 specifically? H100 SXM offers up to 700W of TDP, 3.35TB/s of memory bandwidth, and 900GB/s of NVLink interconnect per GPU, while H100 PCIe is capped at 350 to 400W, roughly 2TB/s of memory bandwidth, and a PCIe Gen5 x16 host link providing 64GB/s in each direction (128GB/s aggregate bidirectional bandwidth), unless paired with a supported NVLink Bridge ^[107].

How does SXM compare to PCIe for the A100? A100 SXM (80GB) supports up to 400W of TDP and 2,039GB/s of memory bandwidth, with NVLink providing 600GB/s of interconnect across up to 16 GPUs via NVSwitch; A100 PCIe (80GB) is capped at 300W and 1,935GB/s of memory bandwidth, with NVLink Bridge kits limited to connecting two GPUs at 600GB/s ^{[57] [50]}.

What are the cooling requirements for SXM GPUs? Cooling requirements depend on the complete server and rack design, not SXM form factor alone. NVIDIA specifies the eight-GPU DGX H100 as air-cooled at 10.2kW maximum, and documents layouts of one system per rack at 10.2kW or two systems per rack at 20.4kW. Higher-density and rack-scale systems can have different requirements; for example, NVIDIA specifies GB200 NVL72 as liquid-cooled ^{[108] [109]}.

Does PCIe GPU performance lag SXM in every workload? No. A single-GPU workload does not benefit from GPU-to-GPU interconnect, but SXM and PCIe versions must still be compared by their published specifications and workload benchmarks. For example, H100 SXM5 has more enabled SMs and FP32 CUDA cores, HBM3 rather than HBM2e, and higher memory bandwidth than H100 PCIe. Multi-GPU workloads add a further advantage for SXM systems through NVLink and NVSwitch ^[66].

Do all cloud providers offer both SXM and PCIe options? No. AWS, Google Cloud, and Microsoft Azure provision their flagship multi-GPU training instances exclusively on SXM hardware, while independent clouds such as Lambda and CoreWeave sell both PCIe and SXM configurations of the same GPU generation side by side ^{[32] [75]}.

When should an organization choose PCIe over SXM? PCIe can be a cost-effective and operationally simpler choice for single- or dual-GPU inference, model fine-tuning that fits in the memory of one or two accelerators, and batch analytical workloads. AON's use of AWS p5.4xlarge supports the narrower observation that a single-GPU workload does not need multi-GPU interconnect; it is not evidence of PCIe form-factor economics. Facilities that cannot support the power and cooling requirements of dense SXM systems should evaluate compatible PCIe offerings and workload needs.

Conclusion

SXM and PCIe are GPU form factors whose available compute configuration, memory technology, power limit, cooling design, and interconnect can differ by product. The first decision question is whether the workload needs high-bandwidth communication among multiple GPUs; if it does, an SXM HGX system with NVLink and NVSwitch can be a strong fit for tightly coupled training workloads. For a single-GPU workload, GPU-to-GPU interconnect is not relevant, but the choice should still be based on the exact GPU specifications, benchmarked performance, availability, price, chassis compatibility, and cooling requirements. AWS's AON example demonstrates the single-GPU point, not PCIe economics, because p5.4xlarge is specified with an H100 GPU using HBM3 memory.

The data gathered for this report, spanning official NVIDIA specifications, cloud provider pricing pages, PCI-SIG's own interconnect standards ^[3], Uptime Institute's facility survey data, AMD's competing OAM format, and five named deployments, points to a durable rather than temporary divide. As NVIDIA expands rack-scale HGX and NVL systems, tightly coupled workloads may increasingly be evaluated as a question of how many GPUs need to act as one and whether the organization's facilities, budget, and workload size justify the required infrastructure. That trend should not be generalized to every Blackwell-based GPU product, because NVIDIA also offers a Blackwell-based PCIe Server Edition. Procurement teams should treat this as a workload-sizing exercise first and a hardware-shopping exercise second: benchmark the actual multi-GPU communication pattern of the target workload before committing to either form factor, since the wrong choice in either direction, over-provisioning SXM for a single-GPU inference service or under-provisioning PCIe for a distributed training job, carries a real and measurable cost.