Out-of-Band Management Network Design for GPU Clusters

The four networks in a GPU cluster

A GPU cluster that loses its out-of-band (OOB) network still trains models, but nobody can power-cycle a hung node or reflash a bad BMC image without walking to the rack. NVIDIA's DGX SuperPOD B300 reference architecture enumerates a compute fabric, an InfiniBand or Ethernet storage fabric, an in-band management network and an out-of-band management network, with the two management networks built on a shared Ethernet fabric "segregated with VXLAN and EVPN" ^[1].

The four planes carry different traffic and fail differently:

The defining property of the OOB plane is in the last column. The NSA/CISA guidance describes BMCs as components that "operate separately from the operating system and firmware to allow for remote management and control, even when the system is shut down" ^[2]. Routing BMC access through the in-band fabric removes that property exactly when it is needed. The high-speed fabrics are covered in the guide on fabric sizing, ports and rails.

What DGX generations do

The split has shifted across DGX generations, mainly in where the in-band ports live and how many BMCs a node carries. Values are copied from the DGX user guides fetched on 2026-10-28 (the H100 and B200 pages show "Last updated on Oct 17, 2026"; the B300 page shows "Last updated on Oct 21, 2026"; none carries a revision number).

Sources: DGX H100 user guide, DGX B200 user guide, DGX B300 user guide.

Three consequences follow. First, every generation keeps a 1 GbE copper BMC port, so the OOB switch remains a commodity 1 GbE access switch. Second, from B200 onward the in-band and storage ports sit on BlueField-3 cards, each with its own BMC. The B300 XDR reference architecture states that "the BMC from both NVIDIA BlueField cards are connected to the out of band Network" alongside the system BMC ^[1]. A DGX B300 therefore needs three OOB copper drops, not one. Third, the B300's on-board 1 GbE host port is OS-owned, and neither the user guide nor the reference architecture pages reviewed here say where it should land. Treating it as in-band (a separate VLAN or VRF from the BMCs) is the conservative choice: a host-OS interface in the BMC segment gives any compromised host a path to every BMC.

OOB topology

The B300 reference architecture uses NVIDIA SN2201 switches for OOB, described as "48 port 1 Gbps Ethernet and 4 x 100 Gbps" ^[3]. The OOB network "is physically rolled up into the aggregation layer (spine layer) of each SU as a dedicated VXLAN" and "carries all IPMI related control traffic" (NVIDIA DGX SuperPOD B300 RA: Network Fabrics, last updated Nov 19, 2025). The XDR variant adds that the OOB network also "serves as the network for fabric management of the compute InfiniBand fabric" (NVIDIA DGX SuperPOD B300 XDR RA: Network Fabrics, last updated Sep 02, 2026). The arithmetic below applies to any 48-port 1 GbE switch.

Port-count formula

For one scalable unit or pod:

• P_node = N x (b + d + h), where N is the number of GPU nodes, b system BMC ports per node, d DPU or SuperNIC BMC ports per node, and h host management ports placed on the OOB switch (0 if they go in-band).
• P_infra = PDU, CDU/cooling controller, fabric switch, storage controller, UFM/management server BMC and console server ports.
• P_total = ceil((P_node + P_infra) x (1 + s)), where s is the spare-port policy.
• Switches = max(ceil(P_total / 48), placement minimum), since cabling policy can force more switches than port count.

Worked example (illustrative)

Assumptions, all illustrative: 64 DGX B300 systems (the SU size in the B300 Spectrum-4 reference architecture, per NVIDIA DGX SuperPOD B300 RA: Key Components); b = 1, d = 2 (two BlueField-3 BMCs), h = 0; 16 racks of 4 systems; 2 metered PDUs per rack; 40 infrastructure management ports for fabric switches, storage, management servers and cooling controllers (a placeholder, not a vendor figure); s = 20%.

1. Node ports: 64 x (1 + 2 + 0) = 192.
2. PDU ports: 16 x 2 = 32.
3. Infrastructure: 40.
4. Subtotal: 192 + 32 + 40 = 264.
5. With 20% spare: 264 x 1.2 = 316.8, rounded up to 317.
6. Port-count minimum: ceil(317 / 48) = ceil(6.6) = 7 switches (336 ports).

The placement policy changes the answer. A switch in every compute rack needs 16 switches for 16 x (12 + 2) = 224 rack-local ports, or 14 of 48 ports used per switch (29%), plus separate switches for the infrastructure rows. One switch per pair of racks needs 8 switches at 28 ports each (58%), with copper runs crossing one rack boundary. Seven central switches minimize hardware but need long copper runs.

Uplink bandwidth is rarely the constraint: 48 x 1 Gb/s = 48 Gb/s of access ports against two 100 Gb/s uplinks is 48:200, or 0.24:1. Steady-state telemetry is small: if, illustratively, 192 BMCs each return a 50 KB sensor payload every 10 seconds, the aggregate is 192 x 50 KB / 10 s = 960 KB/s, about 7.7 Mb/s. The event that matters is mass reimaging via virtual media: illustratively, a 2 GB image to 64 nodes is 64 x 16 Gb = 1,024 Gb, at least 1,024 / 25 = 41 seconds from an image server on one 25 Gb/s link, before BMC-side limits that should be measured rather than assumed.

Addressing follows from the same count: 317 addresses exceed a /24 (254 usable hosts), so either a /23 per SU or separate /24s for node BMCs and infrastructure.

Power and redundancy

The OOB switches should not share fate with the nodes they manage: feed them from a separate or UPS-backed circuit, give each two uplinks to different aggregation switches, and keep a console-server path to the aggregation layer. The DGX guides reviewed here list one 1 GbE BMC interface per system, so a failed OOB access switch isolates every BMC behind it; the per-pair placement above bounds that blast radius to 8 nodes in the worked example.

BMC capabilities

DGX H100 and B200 BMCs list "Redfish, IPMI, SNMP, KVM, and Web user interface" ^[4][5]. The DGX B300 BMC page lists the ports to open on the firewall, which doubles as an inventory of the BMC's attack surface ^[6]:

The exposure column is this guide's suggestion, not NVIDIA's. Preferring Redfish over IPMI is partly protocol security: in IPMI 2.0 RMCP+ RAKP authentication, a remote attacker can obtain password hashes from the BMC's RAKP message 2 response for offline guessing, a specification-level weakness ^[7]. NVIDIA's BlueField BMC documentation (v26.07, updated September 1, 2026) puts it plainly: "IPMI is supported for backward compatibility," and "it is recommended to start new deployments with Redfish only" ^[8].

Redfish for telemetry and provisioning

DMTF's Redfish Specification (DSP0266 version 1.25.0, dated 2026-08-07) defines Redfish as "a standard that uses RESTful interface semantics to access a schema-based data model to conduct management operations," with JSON payloads and OData conventions ^[9]. DMTF lists Release 2026.2 as current ^[10]. Normative details that matter for tooling:

• The service root is /redfish/v1/.
• Implementations "shall support" TLS 1.2 or later, and only cipher suites IANA marks "Recommended."
• HTTP Basic authentication requires HTTPS; session login is a POST to /redfish/v1/SessionService/Sessions, HTTPS only, and suits pollers better.
• Eventing comes in two styles: push (the service POSTs to a subscriber) and server-sent events (SSE) over a long-lived GET.

The Redfish Resource and Schema Guide (DSP2046 2026.2) maps the cluster operations onto standard resources ^[11]:

DGX B300 implements these concretely. Redfish "is enabled by default in the DGX B300 BMC and the BIOS," and NVIDIA documents platform sensors at /redfish/v1/Chassis/DGX/Sensors, GPU tray metrics at /redfish/v1/TelemetryService/MetricReportDefinitions/HGX_PlatformEnvironmentMetrics_0, and virtual media via /redfish/v1/Managers/Self/VirtualMedia/CD_1/Actions/VirtualMedia.InsertMedia ^[12]. NVIDIA's examples use curl -k, which skips certificate validation; production tooling should instead enroll BMC certificates from an internal CA so that -k is never needed.

Redfish telemetry complements in-band GPU telemetry: the BMC sees power, temperatures and firmware state even when the host is down, while DCGM sees utilization, ECC and XID events. The DCGM exporter monitoring and triage guide covers the in-band side.

OpenBMC context

Outside DGX, BMC firmware may be a proprietary vendor stack or derived from OpenBMC, which describes itself as "an open source BMC firmware stack that will work across heterogeneous systems" ^[13]. Its web server, bmcweb, implements "a protocol compliant, DBus to Redfish translator" alongside KVM and serial-console websockets ^[14]. Ask the vendor which stack it ships; it sets patch cadence.

BMC threat model and segmentation

NSA and CISA note that BMC firmware is "highly privileged, executes outside the scope of operating system (OS) controls," and that a compromised BMC can be used to disable TPM or UEFI Secure Boot and manipulate attached storage; EDR and TPM attestation are described as "ineffective at mitigating a compromised BMC" ^[2]. The sheet (June 2023, version 1.0) recommends protecting credentials, VLAN separation, hardened configurations, monthly or quarterly update checks, integrity monitoring, periodic firmware scanning, and "do not ignore BMCs."

Mapped onto a GPU cluster, the controls look like this:

1. No routable exposure. No default route to the internet; if BMCs must fetch updates, permit only update traffic during the window, as NSA/CISA suggest.
2. Allow-listed sources. ACLs on the OOB segment admit only the provisioning servers, the telemetry collector, the fabric manager and a hardened jump host. Tenant and user networks never reach it.
3. Unique credentials. Replace factory credentials before first connection, use unique vaulted per-node passwords, and rotate them on schedule and after staff changes.
4. Protocol minimization. Disable IPMI-over-LAN, UPnP and SLP where tooling allows; restrict SNMP to read-only from monitoring hosts.
5. DPU BMCs count. BlueField-3 BMCs are BMCs: same credential policy, same segment, same patch cadence.

Firmware resiliency

NIST SP 800-193, Platform Firmware Resiliency Guidelines (final, May 2018), is organized around protecting firmware against unauthorized changes, detecting unauthorized changes and recovering "rapidly and securely" ^[15]. The full text lists the BMC as a platform component and assigns these functions to logical roots of trust: a Root of Trust for Update (RTU) "responsible for authenticating firmware updates," a Root of Trust for Detection (RTD) for corruption detection, and a Root of Trust for Recovery (RTRec) for restoring firmware ^[16]. Its authenticated update mechanism "employs digital signatures."

Operationally, this means procurement questions and update policy:

• Ask the vendor what acts as RTU, RTD and RTRec for BMC and BIOS images, and whether a failed BMC flash recovers automatically. The DGX B300's BMC sits on "a new, replaceable DC-SCM" module ^[17], which shortens physical recovery but does not replace an automatic one.
• Stage BMC, BIOS and DPU BMC updates as a canary rack first, then by failure domain, never the whole SU at once.
• Record the pre- and post-update firmware inventory from Redfish for every node, and alert on any version change outside a change window.

Provisioning flow

A repeatable bring-up sequence uses the OOB network for everything that happens before the host OS exists:

1. Inventory as code. Record serial number, rack position, and system and DPU BMC MACs in a version-controlled inventory.
2. DHCP reservations by BMC MAC. Map each BMC MAC to a fixed address and hostname. The DGX B300 guide documents static BMC addressing for networks without DHCP ^[6], but reservations keep the source of truth in one place.
3. Credential and certificate enrollment. Rotate factory credentials and install internal-CA certificates over Redfish.
4. Firmware baseline. Compare FirmwareInventory with the pinned baseline and update via UpdateService until every node matches.
5. OS install. Boot via PXE on the in-band network or via Redfish VirtualMedia.InsertMedia; PXE usually suits mass installs, virtual media single-node recovery.
6. Hand-off to acceptance. Burn-in begins only after inventory, firmware and OOB reachability match the record.

DPU-based management

BlueField devices complicate the plane boundaries. NVIDIA's DOCA documentation (3.5.0) defines DPU mode, in which "the NIC resources and functionality are owned and controlled by the embedded Arm subsystem," and NIC mode, in which "the Arm cores of BlueField are inactive, and the device functions as an NVIDIA ConnectX network adapter" ^[18]. DGX B300 ships BlueField-3 in NIC mode by default ^[17]. In NIC mode the card is an in-band data path plus a BMC on the OOB network. In DPU mode the Arm subsystem is another managed operating system on each card, with its own provisioning, patching and access control. Qualification questions for newer DPUs are covered in the BlueField DPU qualification guide.

Commissioning checklist

The OOB plane should be accepted before the GPU nodes, and its checks belong in the cluster acceptance testing runbook:

The failure drills prove the OOB network stays reachable when everything else is down. GPU Smith can help review an OOB and BMC design against these checks.

Sources

1. DGX H100 User Guide: Introduction to the NVIDIA DGX H100 System - NVIDIA, BMC port and protocols, ConnectX-7 storage/in-band cards, optional management interfaces (accessed 2026-10-28)
2. DGX B200 User Guide: Introduction to the NVIDIA DGX B200 System - NVIDIA, BMC port and protocols, BlueField-3 storage/in-band cards (accessed 2026-10-28)
3. DGX B300 User Guide: Introduction to the NVIDIA DGX B300 System - NVIDIA, BMC on DC-SCM, on-board 1 GbE host port, BlueField-3 NIC mode default (accessed 2026-10-28)
4. DGX B300 User Guide: Using the Baseboard Management Controller - NVIDIA, BMC firewall port table, static BMC addressing (accessed 2026-10-28)
5. DGX B300 User Guide: Redfish APIs Support - NVIDIA, Redfish enabled by default, firmware, telemetry and virtual media URIs (accessed 2026-10-28)
6. DGX SuperPOD B300 Reference Architecture: Network Fabrics - NVIDIA, OOB on SN2201, VXLAN roll-up, IPMI traffic (accessed 2026-10-28)
7. DGX SuperPOD B300 Reference Architecture: Key Components - NVIDIA, SN2201 port configuration, 64-system SU (accessed 2026-10-28)
8. DGX SuperPOD B300 XDR Reference Architecture: Network Fabrics - NVIDIA, network enumeration, BlueField-3 BMCs on OOB, fabric management over OOB (accessed 2026-10-28)
9. Redfish - DMTF, current Redfish release listing (accessed 2026-10-28)
10. DSP0266 Redfish Specification 1.25.0 - DMTF, RESTful/JSON/OData definition, service root, TLS, authentication, eventing (accessed 2026-10-28)
11. DSP2046 Redfish Resource and Schema Guide 2026.2 - DMTF, UpdateService, TelemetryService, VirtualMedia, deprecated Power/Thermal (accessed 2026-10-28)
12. Harden Baseboard Management Controllers - NSA and CISA, BMC threat model and eight recommended actions (accessed 2026-10-28)
13. CVE-2013-4786 - NIST NVD, IPMI 2.0 RAKP password-hash disclosure (accessed 2026-10-28)
14. NVIDIA BlueField BMC Software v26.07: BlueField Management - NVIDIA, last updated September 1, 2026, 1GbE OOB port, Redfish-only recommendation for new deployments (accessed 2026-10-28)
15. OpenBMC - OpenBMC project, self-description (accessed 2026-10-28)
16. openbmc/bmcweb - OpenBMC on GitHub, Redfish translator in bmcweb (accessed 2026-10-28)
17. NIST SP 800-193: Platform Firmware Resiliency Guidelines - NIST, publication status and date, protect/detect/recover (accessed 2026-10-28)
18. NIST SP 800-193 (PDF) - NIST, roots of trust for update, detection and recovery, BMC scope, authenticated updates (accessed 2026-10-28)
19. BlueField Modes of Operation - NVIDIA DOCA 3.5.0, DPU mode, NIC mode, zero-trust mode (accessed 2026-10-28)