InfiniBand vs Spectrum-X vs RoCE vs Ethernet for AI Clusters
InfiniBand vs Spectrum-X vs RoCE vs Ethernet for AI Clusters
Executive Summary
Choosing a network fabric for an artificial intelligence (AI) cluster in 2026 means choosing among four overlapping options: InfiniBand, NVIDIA Spectrum-X (an Ethernet platform), RDMA over Converged Ethernet (RoCE, specifically RoCEv2), and generic or Ultra Ethernet Consortium (UEC) standards-based Ethernet. These are not four independent technologies so much as two link-layer choices (InfiniBand versus Ethernet) crossed with an implementation question (whose RDMA stack and whose silicon). InfiniBand remains a low-latency, deterministic fabric, with switch hop latency in roughly the 200 to 300 nanosecond range against a typical Ethernet hop of 300 to 3,000 nanoseconds [1]. On the June 2026 TOP500 list, InfiniBand-equipped systems rank fifth and seventh; it does not constitute a majority of the top ten ( TOP500. Dell'Oro Group reported that Ethernet switch sales in AI back-end networks "more than doubled and accounted for about two-thirds of data center switch sales in AI clusters during the first quarter 2026" [2], even as InfiniBand sales "more than tripled during the quarter" on the ramp of NVIDIA's Blackwell Ultra platform [3].
Meta built two 24,576-GPU cluster versions for workloads including Llama 3: one uses RoCE over Ethernet and the other uses NVIDIA Quantum-2 InfiniBand. Meta reported ongoing Llama 3 training on the RoCE cluster and later detailed its RoCEv2 operations in a peer-reviewed ACM SIGCOMM 2024 paper [4]. xAI's Colossus supercomputer, which trains the Grok model family, scaled to 100,000 NVIDIA Hopper GPUs on NVIDIA's Spectrum-X Ethernet platform rather than InfiniBand [5], and has since doubled to 200,000 GPUs [6]. Microsoft Azure's ND H100 v5 virtual machines and Oracle Cloud Infrastructure's (OCI) newest Zettascale10 cluster illustrate the split further: Azure still wires each GPU to a dedicated 400 Gigabit-per-second (Gb/s) NVIDIA Quantum-2 InfiniBand connection [7], while Oracle's OpenAI-partnered Stargate site in Abilene, Texas, targeting up to 800,000 GPUs, runs on Oracle's own "Acceleron RoCE networking architecture" [8].
The market context explains why both camps are growing simultaneously rather than one replacing the other. Dell'Oro Group forecasts that spending on data center switches in AI back-end networks will "surpass $100 billion by 2030" [9], while 650 Group projects that the total AI-era data-center-networking market could "approach and potentially exceed $200B" annually by decade's end [10], with AI scale-out Ethernet revenue alone "surging to over $100B by 2030" [11]. The Ultra Ethernet Consortium, founded in 2023 by AMD, Arista, Broadcom, Cisco, Eviden, Hewlett Packard Enterprise (HPE), Intel, Meta, and Microsoft [12], published its 1.0 specification in June 2025 [13] and now counts AMD's Pensando Pollara 400 as the first UEC 1.0-compliant network interface card (NIC) to ship, with Oracle as the first cloud provider to deploy it [14]. For most buyers building multi-thousand-GPU clusters in 2026, the practical decision is not "InfiniBand or Ethernet" in the abstract but a function of tenancy model, in-house networking skills, vendor lock-in tolerance, and how many nanoseconds of tail latency actually change job completion time; this report works through each fabric's architecture, adoption, and quantified trade-offs to make that decision legible, including how the underlying topology, congestion control, and switch-silicon choices interact with each option.
Introduction and Background
Training and serving large-scale AI models depends on moving enormous volumes of data between accelerators, not just on the accelerators themselves. A distributed training job synchronizes gradients across tens of thousands of GPUs after every batch, and Arista's AI networking whitepaper notes that in this "compute-exchange-reduce cycle, approximately 20-50% of the job time is spent communicating across the network," so bottlenecks have "a substantial impact on job completion time" [15]. This is the reason a specialized "back-end" or "east-west" fabric, separate from the front-end network that serves ordinary enterprise traffic, has become a standard feature of large AI clusters. Meta, for instance, explicitly separated GPU-based training into its own backend network when it built its 24,576-GPU clusters [16].
At the center of this back-end fabric is remote direct memory access (RDMA), a mechanism that lets one server's network interface card write directly into another server's memory, bypassing the operating system kernel and the central processing unit (CPU) on both ends. InfiniBand, introduced in the late 1990s, was purpose-built around RDMA and credit-based, lossless flow control from its inception [17]. RoCE, and specifically RoCEv2, retrofits the same RDMA semantics onto standard Ethernet by encapsulating RDMA traffic inside User Datagram Protocol (UDP) and Internet Protocol (IP) packets, which is what allows it to traverse ordinary IP routers and coexist with the rest of a data center's Ethernet infrastructure. NVIDIA Spectrum-X is not a new wire protocol; it is a commercial platform, combining NVIDIA's Spectrum Ethernet switches with its BlueField and ConnectX/SuperNIC network adapters and proprietary RoCE extensions, engineered as a tuned, matched pair to make RoCEv2 Ethernet perform closer to InfiniBand at very large scale [18].
The fourth term in the query, generic "Ethernet," typically refers to the emerging Ultra Ethernet standard: an effort by the UEC to standardize the RoCE-like transport innovations that individual vendors (NVIDIA, Broadcom, AMD, Oracle) have each been building proprietarily, so that AI fabrics can be built from multiple, interoperable vendors rather than one.
The physical topology underneath all four fabrics is usually some variant of a non-blocking Clos or "fat tree" network, in which every leaf (top-of-rack) switch connects to every spine switch so that any GPU can reach any other GPU with the same number of hops and the same available bandwidth, avoiding hot spots. NVIDIA documents that its Quantum-X800 InfiniBand switches "support a two-level, fat-tree topology capable of connecting up to 10,368 network interface cards (NICs) with minimal latency and optimal job locality" [19], while a vendor engineering paper on AI cluster back-end fabrics frames the basic building block of these designs as a "scale unit," a repeatable rack-level module that is replicated to grow the cluster to its final size [20]. Regardless of link-layer choice, most production AI fabrics also implement "rail-optimized" wiring, in which each GPU's dedicated NIC connects to a separate, parallel plane of switches rather than sharing a single uplink, a pattern visible in both NVIDIA's Spectrum-X Multiplane architecture and Oracle's Acceleron design discussed later in this report.
The stakes of this choice are large. Dell'Oro Group's Sameh Boujelbene has framed the shift bluntly: "just as Ethernet prevailed over InfiniBand in large-scale scale-out environments, we are now seeing alternative technologies such as UALink and Ethernet gain momentum in scale-up compute fabrics" [21]. Yet InfiniBand has not disappeared, and in the most recent quarter for which Dell'Oro has published data, InfiniBand sales "more than tripled" while Ethernet accounted for about two-thirds of data center switch sales in AI clusters [3]. The remainder of this report examines each fabric individually, compares them directly on a feature matrix and in published benchmarks, walks through named production deployments, and closes with implications for buyers planning clusters through the rest of the decade.
InfiniBand
Capabilities
InfiniBand is a purpose-built interconnect architecture, standardized by the InfiniBand Trade Association (IBTA), that was designed from the ground up around RDMA and credit-based flow control rather than best-effort packet delivery. Its defining architectural trait is that InfiniBand "prevents packet loss by ensuring receiving buffers have space before transmission begins," using link-level credits rather than the drop-and-retransmit model of conventional Ethernet [22]. NVIDIA's current flagship platform, Quantum-X800, delivers "the world's first end-to-end 800 Gb/s networking" with 144 ports of 800 Gb/s connectivity per switch [23], while the prior-generation Quantum-2 (NDR) platform ships 64 ports of 400 Gb/s in a single 1U chassis with 51.2 terabits per second (Tb/s) of aggregated bidirectional throughput [24]. The IBTA's Volume 1 Release 2.0 specification, published July 31, 2025, supports "XDR speeds up to 200Gb/s per lane, enabling total link bandwidths of 800Gb/s (QSFP) and 1.6Tb/s (QSFP-DD and OSFP)" [25].
Beyond raw bandwidth, InfiniBand includes several features not natively present in Ethernet. Adaptive routing lets switches reroute packets around congested links using real-time queue-depth telemetry rather than a static hash [26]. SHARP (Scalable Hierarchical Aggregation and Reduction Protocol), now in its fourth generation on Quantum-X800, offloads collective operations such as all-reduce directly into switch silicon, and NVIDIA reports service providers seeing "10-20% performance improvements for in-house AI workloads" from SHARP integration with the NVIDIA Collective Communication Library (NCCL) [27]. Quantum-X800 also markets up to 9x higher in-network computing throughput over prior generations, alongside the doubled 800 Gb/s per-port bandwidth described above.
Adoption
InfiniBand's institutional stronghold remains high-performance computing (HPC) and the frontier of AI training. Multiple named hyperscale deployments run on it: Microsoft Azure's ND H100 v5 instances give "each GPU within the VM ... its own dedicated, topology-agnostic 400 Gb/s NVIDIA Quantum-2 CX7 InfiniBand connection" [28], and Azure documented that "the inclusion of NVIDIA Quantum-2 CX7 InfiniBand with 3,200 Gbps cross-node bandwidth ensures seamless performance across the GPUs at massive scale" [29]. GPU cloud provider CoreWeave offers InfiniBand as one of two RDMA-enabled fabrics, describing it as "the fastest, lowest-latency, and most reliable way for GPUs to exchange information in traditional HPC environments" [30]. For historical context, NVIDIA (then Mellanox) reported that its InfiniBand solutions accelerated six of the top ten systems on the June 2019 TOP500 list, including the top three and four of the top five [31]. That historical result should not be read as a current TOP500 share: the June 2026 list places InfiniBand-equipped systems at ranks five and seven ( TOP500.
Strengths and Limitations
InfiniBand's principal strength is a deterministic, low, and well-understood latency floor: switch hop latency in the "~200ns to 300ns" range according to one vendor engineering analysis, versus a considerably wider and more variable range for Ethernet. Its native congestion management and adaptive routing are mature and purpose-built rather than adapted. Its principal limitation is vendor concentration: NVIDIA (through its 2019 acquisition of Mellanox) is effectively the sole supplier of InfiniBand switch silicon and host channel adapters at scale, a dynamic that industry commentary characterizes as "single-vendor pricing" carrying a cost premium; one vendor analysis estimates "the InfiniBand premium typically lands in the +30% to +60% range over open-hardware Ethernet for equivalent capacity" (a claim that should be read as an interested Ethernet-hardware vendor's estimate rather than an audited figure) [32]. InfiniBand also typically requires a dedicated subnet manager and specialized administration skills distinct from a conventional Ethernet network operations team.
NVIDIA Spectrum-X
Capabilities
Spectrum-X is described by NVIDIA as "the world's first Ethernet networking platform for AI," combining the Spectrum-X Ethernet switch with the Spectrum-X SuperNIC to run RoCE at higher effective bandwidth than commodity Ethernet [33]. NVIDIA claims the platform accelerates "AI network performance by 1.6x over off-the-shelf (OTS) Ethernet" [34], a vendor benchmark that should be read alongside independent testing discussed later in this report. Under the hood, Spectrum-X combines the NVIDIA Spectrum-4 (and now Spectrum-6) switch application-specific integrated circuit (ASIC) with the BlueField-3 SuperNIC, ConnectX-7, and newer ConnectX-8 SuperNIC families, and NVIDIA documents that its "advanced RoCE extensions for scalable AI communications" include RoCE adaptive routing, congestion control, and performance isolation designed to hold "up to 95% effective bandwidth across hyperscale systems at load and at scale" [35]. NVIDIA's "Spectrum-X Multiplane" architecture splits each GPU SuperNIC across two or more independent network planes and can scale "Spectrum-X Ethernet networks up to 128K GPUs in two tiers" [36]. In August 2025 NVIDIA added a third tier, Spectrum-XGS Ethernet, a "scale-across" layer for linking multiple physically separate data centers into a single "giga-scale" AI training domain; NVIDIA reports it "delivers up to 1.9x higher NCCL all-reduce bandwidth compared to off-the-shelf Ethernet" for that use case [37].
Adoption
The signature Spectrum-X deployment is xAI's Colossus supercomputer in Memphis, Tennessee. NVIDIA announced that "xAI's Colossus supercomputer cluster comprising 100,000 NVIDIA Hopper Tensor Core GPUs ... achieved this massive scale by using the NVIDIA Spectrum-X Ethernet networking platform" [5], and that the entire facility "achieves unprecedented network performance" with "zero application latency degradation or packet loss due to flow collisions" across all three network tiers [38]. The Register independently confirmed that "unlike most AI training clusters, xAI's Colossus with its 100,000 Nvidia Hopper GPUs doesn't use InfiniBand," describing the choice as notable specifically because it broke with convention [39]. xAI's own site now lists 200,000 H100 GPUs "in a single interconnected cluster," reached in 276 days from groundbreaking [6]. CoreWeave also offers Spectrum-X (RoCE) as one of its two primary HPC interconnect options, alongside InfiniBand, on newer GPU generations such as GB300 [40] and is the launch partner for Spectrum-XGS scale-across networking [41].
Strengths and Limitations
Spectrum-X's core value proposition is capturing much of InfiniBand's tuned, matched-hardware performance while running standard, routable Ethernet frames that integrate with the rest of a hyperscaler's data-center Ethernet fabric, support multi-tenancy, and use SONiC or NVIDIA's Cumulus Linux as an open network operating system [42]. The limitation is that, despite running on Ethernet, Spectrum-X is still a largely NVIDIA-defined, matched-silicon platform (Spectrum switch plus BlueField/ConnectX SuperNIC); it narrows but does not eliminate the vendor-concentration dynamic associated with InfiniBand, since achieving the advertised 1.6x uplift depends on NVIDIA's specific switch-and-NIC pairing rather than being guaranteed on arbitrary third-party Ethernet gear.
RoCE (RDMA over Converged Ethernet)
Capabilities
RoCEv2 encapsulates RDMA transport protocol packets over UDP/IP, allowing GPUs to perform direct memory-to-memory transfers over conventional, routable Ethernet ( NVIDIA documentation. As one technical comparison frames it, "both protocols expose RDMA verbs at the top of the stack... The verb layer is identical. The difference is what happens between the NIC and the wire" [43]. Losslessness on RoCEv2, unlike InfiniBand's native link-level credit system, is achieved through a combination of priority flow control (PFC), a mechanism that pauses specific traffic classes on a link rather than the whole link when a downstream buffer nears capacity, and explicit congestion notification (ECN), which marks packets so endpoints can throttle transmission before buffers overflow, historically managed end to end by Data Center Quantized Congestion Notification (DCQCN). Meta's peer-reviewed SIGCOMM 2024 paper is the most detailed public account of running RoCEv2 at scale: it documents that Meta built its backend fabric "using RDMA Over Converged Ethernet version 2 (RoCEv2) as the inter-node communication transport for the majority of our AI capacity" [44], that this backend fabric "utilizes the RoCEv2 protocol, which encapsulates the RDMA service in UDP packets for transport over the network" [45], and, notably, that Meta "initially attempted to use DCQCN for congestion management but then pivoted away from DCQCN to instead leverage the collective library itself to manage congestion" [46]. Meta reported that after moving to 400 Gigabit deployments, "default DCQCN settings and doubled ECN thresholds compared to 200G networks" degraded performance, and that the firm has since run "over a year of experience with just PFC for flow control, without any other transport-level congestion control," observing "stable performance and lack of persistent congestion for training collectives" [47].
Adoption
RoCEv2 underpins NVIDIA's Spectrum-X and is also used by several large Ethernet AI fabrics. In addition to Meta's RoCE-equipped 24,576-GPU cluster used for ongoing Llama 3 training—the other 24,576-GPU cluster uses InfiniBand [4]—Oracle's OCI Zettascale10 platform, underpinning its Stargate partnership with OpenAI, is "built on next-generation Oracle Acceleron RoCE networking architecture" and provides "ultra‑low‑latency RoCEv2 networking" targeting up to 800,000 GPUs per deployment [48]. Oracle's earlier OCI cluster network used "RDMA over Converged Ethernet (RoCE) on top of NVIDIA ConnectX RDMA NICs" to connect A100 GPU nodes, each with "1.6Tbps (1600Gbps) of full-duplex connectivity" per node [49]. CoreWeave and every Spectrum-X deployment (including xAI's Colossus) also ultimately run RoCEv2 as the wire-level RDMA transport. Amazon Web Services (AWS) takes a related but distinct approach: its Elastic Fabric Adapter (EFA) is "a network device that you can attach to your Amazon EC2 instance to accelerate Artificial Intelligence (AI), Machine Learning (ML), and High Performance Computing (HPC) applications," and AWS states EFA "provides lower and more consistent latency and higher throughput than the TCP transport traditionally used in cloud-based HPC systems," integrating with NCCL and the Message Passing Interface (MPI) rather than exposing raw InfiniBand or RoCEv2 verbs directly to the customer [50].
Strengths and Limitations
RoCEv2's core strength is that it runs on the same Ethernet fabric, tooling, and (largely) the same operator skill set as the rest of a hyperscale data center, avoiding a separate subnet manager and specialized InfiniBand administration. Its core limitation, as Meta's own engineering account illustrates, is that lossless Ethernet requires active, hands-on tuning: PFC alone risks head-of-line blocking, and off-the-shelf congestion control algorithms designed for storage workloads (DCQCN) were found by Meta to behave unpredictably under AI collective-communication traffic patterns at 400G speeds, forcing a bespoke, in-house solution built on "careful coordination between the collective communication library and the network" [51]. A 2022 academic study by researchers from the University of Texas at Austin, Georgia Institute of Technology, and Meta similarly found that "the native RoCE congestion control scheme, based on Priority Flow Control (PFC), suffers from many drawbacks such as unfairness, head-of-line-blocking, and deadlock," which is precisely why the paper set out to evaluate whether general-purpose schemes such as DCQCN, DCTCP, TIMELY, and HPCC actually suit the narrower, more predictable traffic patterns of distributed training collectives [52]. This means RoCEv2's operational simplicity relative to InfiniBand is real but conditional on the scale and sophistication of the operator; Meta's paper is explicit that this tuning took years of iteration.
Standards-Based and Ultra Ethernet
Capabilities
"Ethernet" as a fourth, distinct option in this comparison generally means multi-vendor, standards-based Ethernet running RoCEv2 (or, increasingly, the emerging Ultra Ethernet transport) on switch silicon that is not tied to a single networking vendor's proprietary platform. The switch-silicon layer is dominated by a small number of merchant-silicon vendors. Broadcom's Tomahawk 5, shipping in production volume since March 2023, was "the industry's highest bandwidth switch chip" at launch, delivering "51.2 Terabits/sec of Ethernet switching capacity in a single, monolithic device" and enabling "single-hop connectivity between 256 high-performance AI/ML accelerators, each having 200Gbps of network bandwidth" [53]. Cisco's competing Silicon One G200, also a 51.2 Tb/s device, is "purpose-built" for "high-bandwidth, AI networking, and web-scale switching," offering "deterministic, low-latency, and power-efficient" performance and now underpins Meta's own OCP-inspired Cisco 8501 platform [54] [55]. Arista's Etherlink portfolio delivers "an 800G end-to-end technology platform across front-end, training, inference, and storage networks," supporting cluster sizes "ranging from thousands to 100,000s of XPUs" with one- and two-tier topologies [56].
Layered on top of this silicon, the Ultra Ethernet Consortium, operating under the Linux Foundation, released UEC Specification 1.0 on June 11, 2025, "a comprehensive, Ethernet-based communication stack engineered to meet the demanding needs of modern Artificial Intelligence (AI) and High-Performance Computing (HPC) workloads," covering "all layers of the networking stack, including NICs, switches, optics, and cables, enabling seamless multi-vendor integration" [57]. The specification, updated to version 1.0.2 in January 2026 [58], brings InfiniBand-like transport techniques ("packet spray across all paths, multi-path RDMA, out-of-order delivery, and selective retransmission") to standard Ethernet without requiring single-vendor lock-in [59].
Adoption
AMD's Pensando Pollara 400 shipped as "the industry's first NIC that's compliant with the Ultra Ethernet Consortium's (UEC) 1.0 specification," with Oracle announced as its first cloud-provider customer [60]. AMD followed with Vulcano, an 800 Gb/s, fully UEC 1.0-compliant NIC [61]. Broadcom separately shipped Thor Ultra, which it calls "the industry's first 800G Ethernet NIC" and describes as "fully feature compliant with UEC specification" [62]. The UEC's founding-member roster, AMD, Arista, Broadcom, Cisco, Eviden, HPE, Intel, Meta, and Microsoft [63], grew to include NVIDIA, Oracle, Alibaba Cloud, Baidu, Bytedance, Dell, Huawei, Juniper, Nokia, and Tencent among 97 total member organizations as of August 2024 [64].
Strengths and Limitations
Standards-based Ethernet's core value is supply-chain diversity: buyers can source switch ASICs from Broadcom, Cisco, or Marvell; NICs from AMD, Broadcom, Intel, or NVIDIA; and network operating systems from SONiC, Cumulus, or a vendor's proprietary stack, avoiding dependency on a single company's product roadmap and pricing. Its principal limitation, at least as of mid-2026, is immaturity of interoperability: UEC 1.0 was only ratified in mid-2025, compliant silicon only began shipping in volume through 2026, and (per IBTA's own November 2025 announcement) the InfiniBand and Ethernet RDMA ecosystems still run separate, if increasingly overlapping, interoperability testing (Plugfest) programs [65]. A separate technology worth flagging for completeness, though outside the scope of this report's four named contenders, is Cornelis Networks' revived Omni-Path, originally an Intel design later spun out into an independent company; Network World reports Cornelis "targets price-sensitive deployments where cost-performance optimization matters more than absolute performance" with its CN5000 series aimed at 400 Gb/s AI deployments, illustrating that InfiniBand and Ethernet are not the only two link layers under active development for AI clusters, even if they remain the two that matter for the overwhelming majority of production deployments [66].
Feature Comparison
Table 1 below summarizes how InfiniBand, NVIDIA Spectrum-X, generic RoCEv2 Ethernet, and standards-based/Ultra Ethernet compare across the axes that matter most for AI cluster design: latency, loss handling, multi-tenancy, vendor ecosystem, and typical cost posture.
| Attribute | InfiniBand | NVIDIA Spectrum-X | RoCEv2 (generic Ethernet) | Ultra Ethernet / standards-based |
|---|---|---|---|---|
| Underlying link layer | Native InfiniBand | Ethernet | Ethernet | Ethernet |
| Loss handling | Native credit-based flow control, lossless by architecture [22] | PFC + NVIDIA RoCE congestion/adaptive-routing extensions tuned to Spectrum switches [67] | PFC, with ECN/DCQCN optional; Meta ultimately runs PFC-only with collective-library-driven congestion management | UEC 1.0 defines packet spray, out-of-order delivery, and selective retransmission to reduce PFC dependence [59] |
| Typical switch hop latency | Approximately 200 to 300 nanoseconds [1] | Vendor-optimized, tighter than generic Ethernet but not independently disclosed in nanoseconds | Approximately 300 to 3,000 nanoseconds depending on hardware and mode [68] | Same silicon base as generic Ethernet; UEC aims to close remaining gap |
| Max single-fabric scale (public claims) | Up to 10,368 NICs in a two-level fat tree per switch generation [19] | Up to 128,000 GPUs in a flat, two-tier topology via Multiplane (discussed above) | One 24,576-GPU Meta cluster is deployed with RoCE; Meta's other 24,576-GPU cluster uses InfiniBand [4]; Oracle targets 800,000 GPUs | Not yet demonstrated at comparable single-fabric scale as of mid-2026 |
| Vendor ecosystem | Effectively single-vendor (NVIDIA/Mellanox silicon) | NVIDIA-defined switch and SuperNIC pairing | Broad: any RoCEv2-capable NIC and switch | Open, multi-vendor by design (Broadcom, Cisco, AMD, Arista, Marvell, and others) [69] |
| Illustrative large deployment | Azure ND H100 v5 [7] | xAI Colossus, 100,000 to 200,000 GPUs [6] | Meta's RoCE-equipped 24,576-GPU cluster used for ongoing Llama 3 training [4] | AMD Pollara at Oracle, first UEC-compliant deployment [70] |
The comparison illustrates that the sharpest technical differentiator is still latency and loss-handling determinism, where InfiniBand retains an edge measured in the hundreds of nanoseconds per hop, but that this edge is increasingly outweighed, for hyperscale buyers, by scale, multi-tenancy, and ecosystem considerations that favor Ethernet-family fabrics. Notably, no single fabric wins on every axis: InfiniBand still leads on raw per-hop latency and single-vendor performance tuning, Spectrum-X leads on demonstrated single-fabric GPU count under one commercial platform, and Ultra Ethernet leads on long-term vendor optionality, though it remains the least battle-tested of the four as of July 2026. Buyers evaluating this table should also weigh operational factors that resist tabular summary, such as the depth of in-house Ethernet versus InfiniBand administration expertise, the number of existing vendor relationships, and whether the cluster will ever need to share physical switches with non-AI, multi-tenant workloads, since InfiniBand's subnet-manager model is designed around single-tenant HPC assumptions that do not map cleanly onto multi-tenant public cloud economics.
Performance and Benchmarks
Independent, apples-to-apples benchmarking of InfiniBand against Ethernet-based fabrics is scarce, because most large training runs are proprietary and run on a single fabric rather than being repeated across two for comparison. World Wide Technology (WWT) published a controlled comparison using two 8-way compute nodes and a single-switch network, with matched hardware and the fabric varied. In that limited setup, WWT found that "across generative tests and OEMs, the performance delta between InfiniBand and Ethernet was statistically insignificant (less than 0.03 percent)," that "Ethernet was faster than InfiniBand's best time in three out of nine generative tests," and that "in inference tests, Ethernet averaged 1.0166 percent slower" than InfiniBand [71]. WWT expressly notes that larger-network effects, including oversubscription, contention, and imperfect load balancing, were outside this test's scope; its results therefore do not establish performance equivalence at production scale. WWT frames the broader competitive picture as one in which "the ultra-high-performance domain (perhaps top 3-5 percent of the total market?) still belongs to InfiniBand," while "the vast majority of current InfiniBand deployments can actually be handled by Ethernet" [72].
On the vendor side, NVIDIA's own Spectrum-X benchmarks claim a "1.6x" network performance improvement over off-the-shelf Ethernet and a target of sustaining "breakthrough 95% efficiency across deployments exceeding 100,000 GPUs" [73]; these figures compare Spectrum-X specifically to generic, unoptimized Ethernet, not to InfiniBand, and should be read as a vendor claim rather than an independent measurement. Similarly, Oracle's Acceleron RoCE architecture is described in Oracle's own materials as providing "5 times higher network bandwidth performance" and "up to five times lower network latency compared to competitors" for its original Zettascale cluster [74], reporting "as low as 2 µs (microseconds) latency" for a 131,072-GPU supercluster [75], and Oracle's newer Zettascale10 architecture claims "sub-10 microsecond latency" across the fuller RDMA fabric [76]. These are competitor-facing claims from Oracle rather than third-party audits, and the "competitors" being compared against are not specified in a peer-reviewed setting.
On academic ground, a 2022 study by researchers from the University of Texas at Austin, Georgia Tech, and Meta documented that "the native RoCE congestion control scheme, based on Priority Flow Control (PFC), suffers from many drawbacks such as unfairness, head-of-line-blocking, and deadlock" in general datacenter environments, motivating the design of training-specific congestion control [52]. A November 2025 benchmarking study by Qualcomm researchers concluded that "RoCE can achieve near-linear scaling performance comparable to InfiniBand when properly configured," while cautioning that InfiniBand "imposes significant infrastructure and operational costs" relative to "commodity Ethernet and RDMA" [77]. A June 2026 AMD engineering analysis comparing the network traffic patterns of four production LLM training runs found that "while GPT-4 and DeepSeek-V2 predominantly leveraged InfiniBand for its low-latency guarantees, Llama 3 adopted a hybrid RoCE and IB approach, and Grok 4.0 fully embraced Ethernet-based RoCE through NVIDIA's Spectrum-X platform," concluding that despite the differing fabric choices, "the core traffic patterns ... remain consistent across models, rooted in shared distributed training frameworks such as PyTorch, JAX, and NCCL" [78], a useful reminder that fabric choice is a design decision layered on top of, not a substitute for, the collective-communication software stack. Read together, the evidence shows that properly engineered RoCEv2 can perform closely to InfiniBand in the published small-scale WWT comparison and in the cited configuration-specific research. It does not establish comparable end-to-end training throughput across production-scale fabrics: WWT specifically excluded the larger-network effects that can create differentials. Large deployments of both fabric families demonstrate feasibility at scale, but they are not controlled head-to-head benchmarks. InfiniBand retains a lower, more deterministic latency profile that may matter most for latency-sensitive HPC and frontier-training workloads.
Data Analysis and Evidence
The market data confirms the trend implied by the individual deployment case studies: InfiniBand still leads in dollar terms in some periods and in raw switch-hop latency, but Ethernet-family fabrics (RoCEv2, Spectrum-X, and emerging Ultra Ethernet) now carry the majority of unit volume and are forecast to carry the majority of long-run dollar growth in AI back-end networking.
Table 2 below places five of the named deployments discussed throughout this report side by side, making the relationship between fabric choice and achieved scale concrete before turning to the aggregate market figures that follow. Each figure in this table is drawn from, and separately cited at, the primary source discussed in the corresponding section above.
| Organization / Deployment | Network Fabric | Scale (GPUs or Chips) | Status as of July 2026 |
|---|---|---|---|
| Meta (Llama 3 training cluster) | RoCE over Ethernet | 24,576 GPUs | In production since 2024; Meta's separate 24,576-GPU cluster uses NVIDIA Quantum-2 InfiniBand |
| xAI Colossus | NVIDIA Spectrum-X Ethernet | 100,000 to 200,000 GPUs | In production, still expanding |
| Oracle OCI Zettascale10 / Stargate (with OpenAI) | Oracle Acceleron RoCE (RoCEv2) | Up to 800,000 GPUs targeted per deployment | Announced October 2025; Oracle says the architecture is being deployed at Stargate, while OCI availability was scheduled for the second half of 2026 |
| Microsoft Azure ND H100 v5 | NVIDIA Quantum-2 InfiniBand | Thousands of GPUs per deployment, 3.2 Tbps interconnect bandwidth per virtual machine | Generally available since August 2023 |
| Google TPU v4 supercomputer | Custom optical circuit switching (neither InfiniBand nor Ethernet RoCE) | 4,096 chips | In production since 2020 |
This side-by-side view underscores that scale alone does not predict fabric choice: xAI's operating Colossus deployment uses Ethernet-based RDMA, and Oracle has announced an Ethernet-based RDMA Zettascale10 deployment targeting up to 800,000 GPUs, while Azure's ND H100 v5 keeps InfiniBand as its default. Fabric choice appears to track tenancy model, in-house engineering capacity, and vendor relationship more closely than it tracks raw GPU count.
Dell'Oro Group, a market-research firm specializing in telecommunications and data-center networking, published its most recent quarterly snapshot on June 2, 2026, reporting that "Ethernet switch sales in AI back-end networks more than doubled and accounted for about two-thirds of data center switch sales in AI clusters during the first quarter 2026," even as "InfiniBand sales more than tripled during the quarter, supported by the ramp of 800 Gbps switches shipping with NVIDIA's Blackwell Ultra platform" [79]. Dell'Oro's analysts caution that this InfiniBand rebound may partly reflect "upgrades of the installed base in brownfield deployments rather than greenfield expansion" [80], a distinction that matters for buyers trying to infer future architecture preference from quarterly sales alone. On the vendor-share side of the same quarter, Dell'Oro reported that "Celestica regained the leading position" in Ethernet AI back-end networks, "followed closely by NVIDIA," with "Arista ranked third" and "Cisco recorded the largest share gain during the quarter and ranked fourth" [81]. Also noteworthy: Dell'Oro reported that "800 Gbps switches accounted for the vast majority of the Ethernet switch shipments and revenues in AI back-end networks during the quarter," with "1600 Gbps switches" only beginning to sample and expected to "ramp in the second half of 2026" [82], which places the current generation of both InfiniBand (Quantum-X800) and Ethernet (Spectrum-X, Tomahawk 5/6, Silicon One G200) switch silicon at roughly the same 800 Gb/s per-port speed tier for direct comparison.
Looking further out, Dell'Oro's February 4, 2026 report forecasts that "spending on data center switches deployed in AI back-end networks is forecast to surpass $100 billion by 2030," with "Ethernet ... projected to dominate both the scale-up and scale-out segments of the market" [83]. This is a marked shift from Dell'Oro's July 2024 report, which found that "InfiniBand is currently dominating the AI back-end network market" even while forecasting that "Ethernet is poised to take over soon," in a market then projected to "approach $80 B over the next five years" [84].
Independent researcher 650 Group corroborates the same trajectory. Its January 2026 forecast states that "AI scale-out Ethernet revenue" will surge "to over $100B by 2030," and that, factoring optics, the total data-center-networking market is "poised to approach and potentially exceed $200B" a year by the end of the decade [85]. Separately, 650 Group's scale-up ("inside-the-rack") networking forecast projects PCIe/UALink switching revenue reaching "$3B+," Ethernet-based scale-up at "$8B+," and NVLink at "$25B+" by 2030, indicating that even within GPU-to-GPU scale-up fabrics, where NVIDIA's proprietary NVLink still dominates, Ethernet-based alternatives (branded Scale-Up Ethernet, or SUE) are gaining a material and growing share [86]. An earlier 650 Group forecast from June 2024 had projected the total data-center AI networking market to "surge to nearly $20B in 2025" [87], illustrating how rapidly the addressable market has been revised upward within roughly 18 months.
On the silicon side, Broadcom's Tomahawk 5, the first 51.2 Tb/s Ethernet switch chip, was shown to replace "forty-eight Tomahawk 1 switches in the network, resulting in over 95 percent reduction in power requirements" for equivalent aggregate bandwidth [88], a data point that underscores how much of the AI-networking cost and power story is now driven by switch-silicon generational improvement rather than by the InfiniBand-versus-Ethernet choice per se. Google's decade-long production experience with its Jupiter datacenter network fabric, presented at ACM SIGCOMM 2022 and covering both AI and general cloud workloads, similarly shows how much headroom exists in topology and switching improvements independent of link-layer choice: Google reports that "in this period Jupiter has delivered 5x higher speed and capacity, 30% reduction in capex, 41% reduction in power" through a shift from a traditional Clos fabric to a "direct-connect topology" using optical circuit switches [89]. On adoption breadth, the IBTA reported a "record-breaking Plugfest event that expanded interoperability testing" spanning "both InfiniBand and Ethernet RDMA-based products" in November 2025, a sign that the two ecosystems, while commercially distinct, are increasingly tested and validated together rather than as hermetically separate worlds [90].
Case Studies and Real-World Examples
Meta: RoCEv2 Ethernet at 24,576-GPU Scale for Llama 3
Meta's March 2024 announcement disclosed two 24,576-GPU cluster versions, built on Grand Teton, Open Rack, and PyTorch, for workloads including Llama 3. One uses a RoCE Ethernet fabric based on Arista and OCP rack switches; the other uses NVIDIA Quantum-2 InfiniBand. Meta reported ongoing Llama 3 training on the RoCE cluster, while stating that it had used both cluster types for large GenAI workloads without network bottlenecks [4]. Meta's SIGCOMM 2024 paper remains detailed public evidence that properly engineered RoCEv2 Ethernet can support frontier-scale training, but it is not evidence that both 24,576-GPU Meta clusters use RoCEv2 [91].
xAI Colossus: Spectrum-X Ethernet at 100,000-to-200,000-GPU Scale
xAI's Colossus supercomputer in Memphis, Tennessee, was built to train the Grok family of models and, according to NVIDIA, was constructed by xAI and NVIDIA "in just 122 days, instead of the typical timeframe for systems of this size that can take many months to years," with training beginning just "19 days from the time the first rack rolled onto the floor" [92]. The Register reported the system's peak performance at "98.9 exaFLOPS of dense FP/BF16" before accounting for sparsity [93]. xAI's own site now cites the cluster at 200,000 H100 GPUs, "170PB/s" of aggregate memory bandwidth, and "2.8Tb/s" of per-server network bandwidth for distributed training [94]. Colossus is the clearest public counterexample to the assumption that only InfiniBand can scale to six-figure GPU counts.
Oracle Cloud Infrastructure Zettascale10 / Stargate: RoCE at Multi-Gigawatt Scale
Oracle's OCI Zettascale10, announced October 14, 2025, is described by Oracle as "the largest AI supercomputer in the cloud," connecting "hundreds of thousands of NVIDIA GPUs across multiple data centers" for "up to an unprecedented 16 zettaFLOPS of peak performance," and it is "the fabric underpinning the flagship supercluster built in collaboration with OpenAI in Abilene, Texas, as part of Stargate" [95]. OpenAI's own vice president of Infrastructure and Industrial Compute, Peter Hoeschele, is quoted describing the "highly scalable custom RoCE design" as one that "maximizes fabric-wide performance at gigawatt scale while keeping most of the power focused on compute" [96]. These Oracle and OpenAI statements describe the announced Stargate deployment and its intended RoCE-based architecture; they do not independently verify production availability at Zettascale10's target scale. Oracle also describes the underlying "wide, shallow, resilient fabric" as using "the GPU NIC as a mini-switch and connecting to multiple physically and logically isolated planes," a design intended to boost "scale while reducing network tiers, cost, and power" [97].
Microsoft Azure ND H100 v5: InfiniBand as the Default for Cloud GPU Training
In contrast, Microsoft's ND H100 v5 series, generally available since August 2023, standardizes on InfiniBand rather than Ethernet for GPU-to-GPU communication. Microsoft documents that "ND H100 v5-based deployments can scale up to thousands of GPUs with 3.2 Tbps of interconnect bandwidth per VM," with "each GPU within the VM ... provided with its own dedicated, topology-agnostic 400 Gb/s NVIDIA Quantum-2 CX7 InfiniBand connection" [98]. OpenAI's then-president Greg Brockman is quoted describing the co-design relationship with Azure as "crucial for scaling our demanding AI training needs, making our research and alignment work on systems like ChatGPT possible" [99]. This case illustrates that even as Ethernet-based fabrics gain unit-volume share elsewhere, InfiniBand remains the default architecture for a major hyperscaler's flagship GPU-cloud product line as of the most recent Microsoft Learn documentation update in April 2026 [100].
Google Jupiter and TPU v4: A Non-InfiniBand, Non-Ethernet-RoCE Alternative Worth Noting
Not every large-scale AI cluster chooses between InfiniBand and Ethernet RoCE at all. Google's TPU v4 supercomputer, described in a peer-reviewed ISCA 2023 paper, uses optical circuit switches (OCSes) to dynamically reconfigure its interconnect topology and is explicitly benchmarked by its own authors as much cheaper, lower power, and faster than InfiniBand, with OCSes and underlying optical components representing less than 5 percent of system cost and less than 3 percent of system power, and the TPU v4 supercomputer, deployed since 2020 and outperforming its TPU v3 predecessor by 2.1x, scaled to 4,096 chips sharing 256 tebibytes (TiB) of high-bandwidth memory [101]. The Jupiter network fabric that underlies Google's broader data center estate, including its AI and TPU infrastructure, achieved comparable generational gains through topology engineering rather than a link-layer switch, delivering, as noted above, "5x higher speed and capacity, 30% reduction in capex, 41% reduction in power" over a decade of production operation [89]. This case is included as a reminder that the InfiniBand/Ethernet binary, while dominant for GPU-based clusters, is not the only architecture in production at hyperscale; Google's custom silicon and optical interconnect approach is a genuine third path, albeit one unavailable to buyers who are not Google itself.
Implications and Future Directions
Several structural trends will shape which fabric buyers choose over the next several years. First, the network is proportionally a small but non-trivial share of total cluster cost: one vendor estimate places it at "roughly 5 to 8 percent of AI cluster total cost of ownership over five years," which means fabric choice, while consequential, is rarely the single largest line item in a build [102]. Second, the boundary between "scale-up" (inside a rack, historically NVIDIA NVLink territory) and "scale-out" (between racks, the traditional InfiniBand/Ethernet battleground) is itself becoming contested, as Broadcom's Scale-Up Ethernet (SUE) initiative, now standardized as part of Ethernet for Scale Up Networking (ESUN) with backing from a "who's who group of hyperscalers and vendors" announced at the OCP Summit in October 2025, brings Ethernet-based transport into the scale-up domain that NVLink and AMD's Ultra Accelerator Link (UALink) have historically owned [103].
Third, co-packaged optics (CPO), which integrate optical interfaces directly with switch silicon rather than relying on pluggable transceivers, is expected to reshape both InfiniBand's and Ethernet's power and density profiles; 650 Group projects that "pluggable optics at 800G and 1.6T, plus emerging CPO, could exceed 50% of the annual switch revenue by 2030" [104], and NVIDIA has already begun shipping "co-packaged silicon photonic networking switches" designed "to scale to millions of GPUs" and "multi-site AI factories" on both its Quantum-X800 InfiniBand and Spectrum-X Ethernet lines [105]. Fourth, the frontier of scale is shifting from single-building clusters to multi-site "AI super-factories": NVIDIA's Spectrum-XGS Ethernet, explicitly positioned as a "third pillar of AI computing beyond scale-up and scale-out," anticipates that individual data centers are "reaching the limits of power and capacity within a single facility" and that future frontier training runs will span multiple physically separate sites [106]. Oracle's Zettascale10, targeting multi-gigawatt data center campuses and up to 800,000 GPUs across the Stargate program, is the clearest existing evidence of this shift already occurring in production.
Fifth, buyers should expect networking teams to need broader, not narrower, skill sets over the next several years, since the largest operators increasingly run mixed fleets rather than standardizing on one fabric. CoreWeave, for example, documents that it maintains "InfiniBand and RoCE labels" across its fleet with a shared, "fabric-agnostic schema" so that workload placement and monitoring tooling work identically regardless of which fabric a given node pool uses [107], an operating model that treats fabric choice as a per-cluster procurement decision rather than a company-wide architectural commitment. Finally, the maturation of the Ultra Ethernet Consortium's specification and first compliant silicon (AMD's Pollara and Vulcano NICs, Broadcom's Thor Ultra NIC) over 2025 and 2026 suggests that within the next product cycle or two, buyers will be able to build genuinely multi-vendor, InfiniBand-competitive Ethernet fabrics without depending on any single company's matched-hardware platform, a development that, if it plays out as the UEC's founding members intend, would narrow the remaining rationale for choosing single-vendor InfiniBand to the smallest, most latency-critical segment of the market that WWT and other observers already describe as "top 3-5 percent" of deployments [108].
Frequently Asked Questions (FAQs)
Is InfiniBand better than Ethernet for AI training? InfiniBand offers a lower and more deterministic latency floor, roughly 200 to 300 nanoseconds per switch hop against a wider and more variable range for Ethernet, but independent testing by WWT found the end-to-end performance delta on real generative-AI and inference workloads to be under 1 to 1.02 percent in most cases [71]. "Better" depends on whether the extra nanoseconds and mature tooling justify InfiniBand's single-vendor cost premium for a given cluster's scale and tenancy model.
What is NVIDIA Spectrum-X networking? Spectrum-X is NVIDIA's commercial Ethernet platform for AI, pairing NVIDIA Spectrum switches with NVIDIA SuperNICs and proprietary RoCE extensions to claim "1.6x" performance over generic Ethernet [73]; it powers xAI's 200,000-GPU Colossus cluster [6].
Is RoCE the same as InfiniBand? No. RoCEv2 runs InfiniBand's RDMA transport semantics over standard Ethernet using UDP/IP encapsulation, whereas InfiniBand uses its own dedicated link layer with native credit-based flow control [43].
What is the best network fabric for GPU clusters? There is no single answer independent of scale and organizational context. Deployments in the narrowest, most latency-sensitive tier of scale still favor InfiniBand, while Meta and xAI demonstrate that RoCEv2-based Ethernet can support large training clusters. Oracle has announced an Acceleron RoCE deployment for its OpenAI-partnered Stargate site, targeting up to 800,000 GPUs [109].
Is the Ultra Ethernet Consortium meant to replace InfiniBand? UEC's own founding framing is to standardize an "Ethernet-based, open and interoperable, high-performance, full-communications stack architecture" for AI and HPC, explicitly to avoid the vendor lock-in associated with single-vendor fabrics [110], which functionally positions it as a long-term, multi-vendor alternative to InfiniBand rather than a modification of InfiniBand itself.
How much does an AI network fabric cost relative to the rest of the cluster? Estimates place networking at roughly 5 to 8 percent of five-year total cost of ownership for an AI cluster, with InfiniBand carrying, per one Ethernet-hardware vendor's estimate, a 30 to 60 percent premium over open-hardware Ethernet for equivalent capacity [32]; this figure should be treated as indicative rather than audited.
Does AWS use InfiniBand or Ethernet for AI training? AWS's Elastic Fabric Adapter runs over the company's own Ethernet-based Scalable Reliable Datagram transport rather than native InfiniBand, and AWS documents that it "supports RDMA (Remote Direct Memory Access) write on most supported instance types that have Nitro version 4 and later," integrating with NCCL and MPI so that customers do not need to manage InfiniBand-specific tooling directly [111].
Can InfiniBand and Ethernet coexist in the same organization? Yes, and this is increasingly the norm rather than the exception among GPU cloud providers; CoreWeave, for instance, operates both fabrics side by side, applying a shared labeling schema so that InfiniBand and RoCE node pools are managed with the same tooling [107].
Conclusion
The choice among InfiniBand, NVIDIA Spectrum-X, RoCEv2, and standards-based or Ultra Ethernet for an AI cluster is not a single binary decision but a layered one: link layer (InfiniBand versus Ethernet), then, if Ethernet, whose matched-hardware platform (NVIDIA's Spectrum-X) versus whose open, multi-vendor stack (generic RoCEv2 today, Ultra Ethernet increasingly going forward). InfiniBand remains a purpose-built, low-latency option and continues to anchor major cloud GPU products such as Microsoft Azure's ND H100 v5 series; on the June 2026 TOP500 list, InfiniBand-equipped systems rank fifth and seventh ( TOP500. But the market has clearly moved. Ethernet switch sales in AI back-end networks accounted for about two-thirds of data center switch sales in AI clusters in the first quarter of 2026, underpin frontier training clusters at Meta and xAI and an announced Oracle Stargate deployment with OpenAI, and are forecast by independent analysts to dominate long-run dollar growth in AI networking spending through the end of the decade.
For organizations building or buying AI cluster infrastructure, the practical framework that emerges from this evidence is straightforward. Choose InfiniBand when the workload sits in the narrow, latency-critical tier that WWT estimates at roughly the top 3 to 5 percent of deployments, when the cluster is single-tenant, and when a single-vendor operational model is acceptable in exchange for a mature, deterministic fabric. Choose NVIDIA Spectrum-X when the buyer wants Ethernet's routability and multi-tenancy with NVIDIA's tuned, matched-hardware performance uplift, and is comfortable staying within NVIDIA's switch-and-SuperNIC ecosystem. Choose generic RoCEv2 Ethernet, following the model Meta has published in detail, when in-house Ethernet operations expertise exists and the organization is prepared to invest in the congestion-control and routing tuning that hyperscale RoCEv2 deployments require. And watch Ultra Ethernet closely: with its 1.0 specification ratified in mid-2025 and updated through version 1.0.3 in July 2026, with AMD's Pollara announced for deployment by Oracle as its first cloud-provider customer and Broadcom's Thor Ultra shipping separately, it is the technology most likely to determine whether the AI networking market of the early 2030s looks like today's InfiniBand-dominated HPC world, today's NVIDIA-dominated Spectrum-X world, or a genuinely open, multi-vendor Ethernet market that the UEC's founding members explicitly set out to build.
External Sources
About GPUSmith
GPU Smith is an independent engineering firm that specifies, procures, integrates and validates private AI compute infrastructure on Nvidia reference architectures, from a single inference node to multi-megawatt compute halls. Every engagement is delivered against written acceptance criteria and an as-built documentation set, with procurement at a disclosed margin and no reseller quota or cloud of its own. Six disciplines: hardware integration and commissioning; cluster architecture and sizing; inference build-out; serving optimization; datacenter operations; and sovereign/air-gapped systems. Core thesis: at sustained load, the amortized cost of owned hardware falls below per-token cloud and API pricing, and GPU Smith locates that crossover for a defined workload and states build/no-build in writing. Sectors served: government and regulated enterprise (bounded inference), scaling AI teams past the ownership crossover, and investors/operators needing technical due diligence.
DISCLAIMER
This document is provided for informational purposes only. No representations or warranties are made regarding the accuracy, completeness, or reliability of its contents. Any use of this information is at your own risk. GPUSmith shall not be liable for any damages arising from the use of this document. This content may include material generated with assistance from artificial intelligence tools, which may contain errors or inaccuracies. Readers should verify critical information independently. All product names, trademarks, and registered trademarks mentioned are property of their respective owners and are used for identification purposes only. Use of these names does not imply endorsement. This document does not constitute professional or legal advice. For specific guidance related to your needs, please consult qualified professionals.