DAC, AOC, LPO and CPO: Choosing Optics for AI Fabrics

Why optics are now a power and reliability line item

In a scale-out GPU fabric each GPU typically owns one NIC port, and each port starts a chain of transceivers and fiber through two or three switch tiers. NVIDIA's 800G twin-port OSFP transceivers for Quantum-2 and Spectrum-4 switches are rated at 17 W maximum; a single-port 800G XDR DR4 module is rated at 18 W maximum. NVIDIA itself describes pluggable optics as drawing "often 30W per interface" in the post that launched its co-packaged switches. Across a few thousand ports that is tens of kilowatts.

They are also a failure source. In Meta's account of a 54-day Llama 3 405B pre-training snapshot on up to 16K H100 GPUs, "Network Switch/Cable" accounted for 35 of 419 unexpected interruptions, or 8.4%. That category mixes switches and cables, so it is not a transceiver failure rate, but every added module and connector is another thing that can flap a link and stop a synchronous job.

The media decision is therefore made per link tier, against three constraints: reach, watts per port, and how a failure gets repaired. For the fiber plant itself (polarity, loss budgets, end-face inspection) see our fiber cabling best practices; for port counts per tier see fabric port and rail sizing.

Form factors: OSFP, QSFP112 and twin-port OSFP

Three pluggable form factors dominate NVIDIA-based AI fabrics, and they are not interchangeable.

• QSFP112 is "a single-port, 4-channel, 400G" form factor used on ConnectX-7/QSFP112 adapters and BlueField-3 DPUs. The same page notes that "BlueField-3 DPUs only use the QSFP112."
• Single-port OSFP (4-channel, 400G at 100G-PAM4 or 800G at 200G-PAM4) appears on adapters with a flat top ("riding heat sink", RHS), because the cooling is provided by the NIC cage. NVIDIA lists a "400G RHS, single-port, 4-channel, OSFP for ConnectX-7/OSFP network adapters" ^[1], and its 800G MMS4A20 is a flat-top OSFP intended for "Linking ConnectX-8 cards with Quantum-X800 QM3x00 Switches".
• Twin-port OSFP packs "two 400Gb/s engines inside the transceiver that exit to two 400G optical or copper ports" ^[1]. This is why a Quantum-2 QM9700 offers "64 ports of NDR 400Gb/s" from 32 OSFP interfaces, and why a Quantum-X800 Q3400 offers "144 XDR ports over 72 OSFP cages". The switch-side version has integrated cooling fins: "Extra cooling fins are used on top to support 17-Watt transceivers", and Quantum-2 and Spectrum-4 air-cooled switches "require finned-top OSFP shells".

The practical consequences:

1. Flat-top and finned modules are not cross-compatible mechanically, even at the same data rate. Order switch-side and NIC-side SKUs separately.
2. Adapter power is quoted without optics. ConnectX-7 typical power is given as, for example, 25.9 W for the single-port OSFP MCX75310AAC-NEAT and 24.9 W for the single-port QSFP112 MCX715105AS-WEAT, both labelled "Typical power with passive cables in PCIe Gen 5.0 x16". Budget the module on top.
3. Cage power limits differ by SKU. Some cards list per-port module limits (5.1 W "thermally supported" per QSFP112 port on the MCX713106A*-VEAT). Check before specifying a 9 W module.

Passive DAC and copper limits at 200G-PAM4

Passive direct-attach copper (DAC) is the cheapest, lowest-power medium: nothing in the cable is active. At 100G-PAM4 per lane (NDR InfiniBand, 400GbE/800GbE on 112G SerDes), NVIDIA documents DAC "reach is up to 3 meters," with power "<0.1 Watts," using 30AWG wire below 2 m and 26AWG up to 3 m (LinkX 100G-PAM4 overview, last updated 2025-11-18). The same generation adds linear active copper (LACC), "a DAC cable that includes linear amplifier signal booster ICs (not a DSP)" that "Extends the reach to 3, 4, and 5-meters" at "1.5 Watts on 800G end and 0.6W and 0.35W for split ends."

At 200G-PAM4 per lane the copper budget shrinks. The IEEE P802.3dj objectives for 200G, 400G, 800G and 1.6T Ethernet set the twin-axial copper target as "a reach of up to at least 1.0 meter" (objectives as adopted 14 March 2024), versus the 2-3 m that 100G-lane DAC delivered in practice. NVIDIA's consolidated XDR parts list is now marked "deprecated" in favour of per-product pages. One of them, the MCA4K50 1.6T OSFP active copper cable for XDR switch-to-switch links (8 x 200G-PAM4), comes in 1.1, 1.5, 2, 2.5 and 3 m lengths with "Max power consumption 27 Watt" ^[2]. At 200G per lane, 3 m of copper therefore means an active cable drawing up to 27 W, not a passive DAC under 0.1 W.

Active electrical cables (AEC) put a retimer DSP in each connector to recover reach. Credo's 2021 800G AEC, for example, was an "8 x 112G per lane copper cable interconnect" available in "lengths up to 2.5m," with a vendor claim of half the power of optical cabling ^[3]. AECs trade some of DAC's power advantage for extra metres and add active silicon that can fail.

Engineering takeaway: at 200G per lane, plan copper only for links that stay inside a rack or between adjacent racks, and confirm the reach of the exact part, not the generation.

AOC vs pluggable optics

An active optical cable (AOC) has its transceivers permanently attached: no separate fiber plant, but the whole assembly is the replaceable unit and its length is fixed at order time. In NVIDIA's NDR-generation line, AOCs survive only as a compatibility bridge: they are "offered for backwards compatibility with 2x200G twin-port OSFP" and are "the only cables that can link NDR/400GbE to HDR/200GbE systems".

For new 400G/800G fabrics the default is a separable pluggable transceiver plus structured fiber. Reach classes in the NVIDIA NDR line are:

At 200G-PAM4 (XDR, 800G per port) the single-mode options documented by NVIDIA include the MMS4A20, an 800G single-port DR4 OSFP with "Maximum reach: 500-meters," "18W max power," "2W low-power mode," a 1310 nm laser and a "Single MPO-12/APC optical connector" over 9 µm single-mode fiber ^[4], and the MMS4C10-XM800, described as using "4-channels of 200G-PAM4 electrical modulation" with "16W max power" and a 500 m reach that "assumes two optical patch panels or four optical connectors in the link" ^[5]. The MMS4C10 page title still says twin-port 2xDR4 with two MPOs, but its feature list (four 200G-PAM4 lanes, one MPO-12/APC) is the self-consistent reading for an 800G DR4 module, and we follow it; confirm connector count with your supplier before ordering trunks.

If your cabling has more mated pairs than the datasheet assumes, recompute the loss budget.

DSP-based pluggables vs linear pluggable optics (LPO)

A conventional pluggable contains a DSP that retimes and equalizes the electrical signal from the host. Linear pluggable optics remove it. The LPO MSA describes the approach as "removing the digital signal processing (DSP) function from the pluggable optical module", claiming lower power, lower latency and lower module cost versus DSP-based modules. In March 2025 the MSA released a specification for "100 Gbps per lane linear pluggable optics" targeting "up to 800 Gigabit Ethernet," stating that it "has been validated by extensive member interoperability testing." The MSA site describes work "extending to 200 Gb/s per lane" ^[6]; its member list includes Arista, Broadcom, Cisco, Nvidia and others.

The DSP does not disappear; its job moves to the host. Semtech, a component vendor with a stake in the outcome, puts it plainly: LPO "move[s] the equalization burden from the module back to the host switch SerDes" ^[7]. Its 200G-per-lane power figures are useful as an order of magnitude, but they are a vendor's targets:

Semtech also says LPO is not a universal replacement: links with "high-loss electrical channels, significant reflection environments" may be better served by RTLR or fully retimed modules.

Interoperability caution. An LPO link is qualified as a specific host SerDes + host channel + module combination; a DSP module masks host-channel differences, an LPO module does not. Require the switch or NIC vendor to list the exact LPO SKUs they support. We found no independent 200G-per-lane LPO interoperability results as of 2026-10-28.

Co-packaged optics: what NVIDIA and Broadcom claim

Co-packaged optics (CPO) move the optical engines next to the switch ASIC, eliminating the trace to the front-panel cage and the module DSP. NVIDIA's August 2025 post says this cuts electrical loss from "up to 22 dB for 200 gigabit-per-second channels" to "only ~4 dB," and power from "often 30W per interface" to "as low as 9W" ^[8]. The announced products:

Source: NVIDIA developer blog; Spectrum-X availability wording from NVIDIA's silicon photonics page (2026-10-28). The same post claims "3.5x" power efficiency, "10x" resiliency "from fewer optical components that can fail," and "1.3x" faster time-to-turn-on versus pluggable architectures. These are vendor claims without a published methodology.

NVIDIA's XDR switch manual lists the CPO model as the Q3450-LD, with "144 800Gb/s ports connected via direct MPO connectors" and no pluggable transceivers. Lambda, which had early access, described it to SDxCentral as a 4U chassis with "115.2 terabits per second (Tb/s)" of capacity, "18 removable light-source modules" feeding the MPO ports and "four UDQ4 liquid cooling connections" ^[9]. Lambda's estimate in the same piece is that a 128k-GPU cluster would need "5 million lasers" with traditional optics versus "1 million" with CPO, i.e. a 5x reduction, and it cited 25 W per transceiver in a conventional 72-port switch. That article describes the hardware as in "full production" ahead of a release "later this year". As of 2026-10-28, NVIDIA's silicon photonics and Quantum-X800 pages give no Quantum-X Photonics availability date or shipping status.

On the Ethernet merchant-silicon side, Broadcom reported "one million cumulative 400G equivalent port device hours of flap-free CPO operation at Meta," in Meta's "high-temperature lab characterization environment," and a 65 percent reduction in optics power versus pluggable modules ^[10]. That is the closest thing to field data in public, but it is a vendor press release summarizing a lab characterization, not an independent study.

Resiliency and serviceability tradeoffs of CPO

CPO changes what fails and how you fix it.

• Fewer parts, fewer flaps (claimed). No pluggable module or front-panel electrical connector per switch port; NVIDIA's "10x resiliency" claim and Broadcom's flap-free hours rest on this.
• Lasers stay removable. The Q3450-LD's 18 removable light-source modules feed 144 ports, so each serves several ports. The sources above do not say what one light-source failure takes down or whether it can be replaced live; ask.
• The optical engine is not a pluggable. With pluggables a bad port is a module swap. With CPO, a failed optical engine (not a laser) is repaired at whatever the field-replaceable unit is, which can be larger than one port; get this in writing.
• Only the switch side changes. NIC-side optics remain pluggable.
• Liquid cooling. NVIDIA's announced CPO switches are liquid-cooled, so network rows inherit the facility-water and leak-detection regime of the GPU racks.

Decision matrix by tier

The matrix is our engineering judgment, not a vendor recommendation. For how these tiers map to InfiniBand, Spectrum-X and generic RoCE fabrics, see InfiniBand vs Spectrum-X vs RoCE.

Power and cost worked example

All arithmetic below is ours, using maximum (not typical) module ratings, so it is an upper bound.

Scenario. 1,024 GPUs, one 400G NDR NIC port per GPU, two-tier non-blocking fat tree on 64-port QM9700-class switches (32 OSFP cages, twin-port modules). Every link is optical.

Switch counts:

• Leaves: 32 downlinks + 32 uplinks each. 1,024 / 32 = 32 leaves.
• Spines: 32 x 32 = 1,024 uplinks / 64 ports = 16 spines.

Module counts and power:

Per GPU: 35,328 / 1,024 = 34.5 W of optics, and 2,560 / 1,024 = 2.5 modules. Annual energy at full rating: 35.328 kW x 8,760 h = 309.5 MWh before cooling overhead. At an assumed $0.10/kWh that is about $30,950 per year in direct electricity, before PUE.

Copper in the first tier. If leaves sit in the server rack so NIC-to-leaf links fit passive DAC (<0.1 W), the NIC-side and leaf-downlink optics disappear: 9,216 + 8,704 = 17,920 W, about half the budget. Leaf placement is an optics power decision.

CPO sensitivity at 800G. Repeat the topology at 800G per GPU on 144-port Q3400-class switches. Switch-side 800G ports = leaf down + leaf up + spine down = 3 x 1,024 = 3,072, carried in 1,536 twin-port cages. NIC-side MMS4A20 modules: 1,024 x 18 W = 18,432 W.

NVIDIA rates its 1.6T twin-port XDR transceiver, the MMS4A00 (2xDR4, 500 m), at "33.5W max power" ^[11]. For CPO we use NVIDIA's "as low as 9W" per interface and treat each 1.6T cage as one interface (an assumption):

• Pluggable: 1,536 x 33.5 W = 51,456 W
• CPO: 1,536 x 9 W = 13,824 W
• Difference: 37,632 W, roughly 37 W per GPU

The NIC-side 18,432 W does not change under CPO. If NVIDIA's 9 W means one 800G port rather than a 1.6T cage, the CPO figure doubles to 27,648 W and the saving falls to 23,808 W.

Acceptance testing optics

Marginal optics pass a ping test and fail under load. Minimum gates:

1. Inventory match. Read each module's EEPROM and compare part number and firmware to the bill of materials. On NVIDIA hardware, mlxlink --cable --dump dumps cable pages and mlxlink --cable --ddm returns "Digital Diagnostic Monitoring information" ^[12].
2. DDM baselines. Record per-lane temperature, Tx bias, Tx and Rx optical power at commissioning; later drift is the early warning.
3. BER under traffic. mlxlink -c shows "Physical Counters and BER Info"; module PRBS can be run with --cable --prbs_mode on the media or host side ^[12]. Set pass criteria on pre-FEC BER margin, not only on zero uncorrectable errors.
4. Flap counting. Zero link-down events over a burn-in window under all-to-all load.
5. End-face inspection and cleaning on every MPO before mating, especially APC single-mode.

Detailed procedures are in our RoCE acceptance testing runbook and InfiniBand fault isolation guide.

What to ask vendors

• Exact transceiver and cable SKUs per tier, with max power, reach and the number of mated connector pairs the reach assumes.
• Finned (IHS) vs flat-top (RHS) shell for each position; cage power limit on every NIC SKU.
• For 200G-per-lane copper: the datasheet reach of the specific DAC, LACC or AEC part, and whether it is supported switch-to-NIC or switch-to-switch only.
• For LPO: the list of qualified host platforms and module SKUs, and who owns a link that fails BER.
• For CPO: the field-replaceable unit for a failed optical engine; how many ports one light-source module serves; whether light-source modules are hot-swappable; required facility water temperature and flow; current general-availability date in writing.
• Spares ratio and RMA turnaround for modules, AOCs and light sources.

GPU Smith can review these media choices against a specific fabric design.

Sources

1. NVIDIA Technical Blog, "Scaling AI Factories with Co-Packaged Optics for Better Power Efficiency," 2025-08-18. https://developer.nvidia.com/blog/scaling-ai-factories-with-co-packaged-optics-for-better-power-efficiency
2. SDxCentral, "Lambda offers glimpse into Nvidia's CPO switch," 2026-06-02. https://www.sdxcentral.com/news/lambda-offers-glimpse-into-nvidias-cpo-switch-enabling-more-tokens-through-substantial-power-savings/
3. NVIDIA, MMS4A20 800Gbps DR4 single-port OSFP transceiver. https://networking-docs.nvidia.com/9iat0mosfp800sprhs
4. NVIDIA, MMS4C10-XM800 800Gbps 2xDR4 transceiver. https://networking-docs.nvidia.com/mms4c1x800/
5. NVIDIA, MMS4X00-NM 800Gbps twin-port OSFP 2xDR4 transceiver. https://networking-docs.nvidia.com/mms4x00nm800g500m/
6. NVIDIA, ConnectX-7 hardware specifications. https://networking-docs.nvidia.com/connectx7hw/specifications
7. NVIDIA, LinkX 100G-PAM4 Product Line Overview (last updated 2025-11-18). https://docs.nvidia.com/networking/display/400g100gpam4ovdev/LinkX-100G-PAM4-Product-Line-Overview
8. NVIDIA, 800Gbps XDR (200G-PAM4) Transceivers, Copper Cables, and Fiber Parts List (deprecated notice). https://docs.nvidia.com/networking/display/800gbps-xdr-200g-pam4-transceivers-copper-cables-and-fiber-parts-list.pdf
9. NVIDIA, Q32xx and Q34xx XDR 800Gb/s InfiniBand Switch Systems User Manual, Introduction. https://networking-docs.nvidia.com/xdrswitcheshw/introduction
10. NVIDIA, QM97x0 NDR InfiniBand Switch Systems User Manual, Introduction. https://networking-docs.nvidia.com/qm97x0hw/Introduction
11. IEEE P802.3dj Task Force, Adopted Objectives, 2024-03-14. https://www.ieee802.org/3/dj/projdoc/objectives_P802d3dj_240314.pdf
12. Lightwave, "LPO MSA releases linear pluggable optical modules specification," 2025-03-31. https://www.lightwaveonline.com/home/article/55278631/lpo-msa-releases-linear-pluggable-optical-modules-specification
13. LPO MSA home page. https://www.lpo-msa.org/
14. Semtech blog, "AI Data Center Basics: What Is Linear Pluggable Optics (LPO)," 2026-08-04. https://blog.semtech.com/ai-data-center-basics-what-is-linear-pluggable-optics-lpo
15. Broadcom, "Broadcom Showcases Industry-Leading Quality and Reliability of Co-Packaged Optics," 2025-10-01. https://investors.broadcom.com/news-releases/news-release-details/broadcom-showcases-industry-leading-quality-and-reliability-co
16. Llama Team, AI @ Meta, "The Llama 3 Herd of Models," arXiv 2407.21783 (v3, 2024-11-23). https://arxiv.org/html/2407.21783v3
17. Converge Digest, "Credo announces 800G HiWire Active Electrical Cables," 2021-10-31. https://convergedigest.com/credo-announces-800g-hiwire-active/
18. NVIDIA Firmware Tools (MFT) 4.37.0, mlxlink utility. https://networking-docs.nvidia.com/mftswum/4.37.0/mlxlink-utility
19. NVIDIA, MMS4A00 1600Gbps 2xDR4 twin-port OSFP transceiver (last updated 2026-10-14). https://networking-docs.nvidia.com/9iahx00xmosfptcvr1600
20. NVIDIA, MCA4K50 1600Gbps OSFP active copper cable, ordering information and key features (last updated 2026-07-20). https://networking-docs.nvidia.com/mca4k50osfp1600/ordering-information
21. NVIDIA, Silicon Photonics networking page (accessed 2026-10-28). https://www.nvidia.com/en-us/networking/products/silicon-photonics/