RoCE Fabric Acceptance Testing: PFC, ECN and CNP Runbook

Executive Summary

A RoCE fabric acceptance test should prove a bounded proposition: the exact hosts, network interface cards (NICs), links, switch application-specific integrated circuits (ASICs), network operating system (NOS) builds, firmware, and quality-of-service (QoS) policy under test behave as the approved design specifies. It cannot prove that every future workload or failure will be harmless. The evidence must therefore identify versions, topology, maximum transmission unit (MTU), trust mode, Differentiated Services Code Point (DSCP) or Priority Code Point (PCP), traffic-class mappings, Priority Flow Control (PFC), Explicit Congestion Notification (ECN), and Congestion Notification Packet (CNP) handling at every hop. NVIDIA's current documentation is pinned to Cumulus Linux 5.18 ^[1], while its generated RoCE settings are ASIC-specific ^[2]. These are examples for that release and platform, not universal settings.

The correct validation sequence is freeze, baseline, load, correlate, restore. First prove routing or VLAN reachability, end-to-end MTU, RoCEv2 classification on registered UDP port 4791 ^[3], and consistent priority mapping. Then capture synchronized idle counters on every ingress, egress, and host. Apply a controlled ladder: single flow, message-size sweep, externally orchestrated many-to-one incast, bidirectional traffic, and the target collective. Perftest can report named sysfs counter deltas ^[4], while NCCL Tests checks both collective performance and correctness ^[5]. Synthetic and collective tests answer different questions.

Interpret deltas, direction, priority, and timing, not isolated snapshots. ECN marks show a queue signaled congestion; RFC 3168 permits marking instead of dropping an ECN-capable packet ^[6]. The receiver then returns a CNP toward the sender, and the sender's reaction point should throttle. PFC is one-hop, priority-scoped backpressure, not end-to-end rate control. Linux exposes receive pause duration in microseconds per priority ^[7]. A rising switch no-buffer discard, NIC buffer discard, or host out-of-buffer counter is loss evidence, not a healthy congestion signal.

Acceptance requires buyer-supplied limits for zero or permitted discards, pause duration, latency, throughput, and collective completion. No defensible universal percentage exists. The evidence pack should contain raw before and after counters, timestamps, offered load, path, topology, configs, software inventory, artifacts, pass criteria, exceptions, and rollback state. GPU Smith's stated practice is to deliver engagements against written acceptance criteria and an as-built record ^[8]; that adjacent engineering posture is useful here because the result should remain vendor-version-pinned and reproducible.

Introduction and Background

Remote Direct Memory Access over Converged Ethernet version 2 (RoCEv2) carries InfiniBand transport over User Datagram Protocol (UDP), allowing Layer 3 routing while the adapter performs the data movement. NVIDIA describes the encapsulation directly as InfiniBand transport over UDP ^[9]. Fast data movement does not remove Ethernet congestion. Instead, a production design typically combines queue classification, ECN and endpoint rate response, with PFC only where the design calls for a lossless class.

The central acceptance question is not whether packets pass at idle. It is whether the intended data and feedback traffic remain correctly classified and observable while queues grow under representative contention. ECN, CNP, and PFC are different signals. ECN marks a packet at a congested queue. The receiver observes Congestion Experienced and sends a CNP back to the source. PFC tells a directly attached transmitter to stop one priority temporarily. IEEE defines PFC as per-traffic-class control on full-duplex links ^[10], and identifies the class through VLAN priority ^[11]. PCP is a three-bit 802.1Q field ^[12].

This report is an operational RoCE network validation checklist, RoCE PFC testing procedure, RoCE ECN validation, and RoCE CNP testing guide. It also covers RoCE buffer counter monitoring, PFC pause frame counters RoCE operators need, a repeatable data center RoCE acceptance test, and evidence-based RoCE congestion management testing. The method is vendor-neutral at the decision level, but every actual command and expected counter must be pinned to the installed platform.

What a RoCE Acceptance Result Can Prove

An acceptance result proves only what its controlled boundary supports. Record the boundary before the first packet is generated. At minimum, the manifest should include:

• Topology: host, NIC port, cable, leaf, spine, peer leaf, destination port, and Equal-Cost Multipath (ECMP) path selection.

• Switch identity: manufacturer, model, ASIC generation, NOS image, boot image, active QoS profile, and configuration checksum.

• Host identity: operating system, kernel, RDMA core, driver, firmware, PCIe location, NUMA relationship, and active RoCE mode.

• Packet contract: VLAN, source and destination address, UDP destination port, MTU, DSCP, PCP, service class, traffic class, and queue. RFC 2474 requires codepoint-to-per-hop-behavior mappings to be configurable ^[13].

• Congestion contract: lossless or lossy mode, ECN method, PFC priorities, CNP class and scheduling, headroom, shared-pool policy, and watchdog behavior.

• Success contract: offered load, test duration, message sizes, fan-in, permitted loss, latency distribution, throughput range, pause budget, and collective completion criteria.

Version capture is functional evidence, not paperwork. Upstream ethtool -i queries the driver associated with a network device ^[14]; NVIDIA documents ofed_info -s for the installed MLNX_OFED version ^[15]. Counter names are not portable: ethtool notes little commonality in vendor-specific statistic names ^[16].

A passing run does not establish a universal ECN threshold, a generic safe PFC rate, or immunity to all traffic patterns. It does establish reproducible behavior for the recorded build. If firmware, cabling, MTU, topology, QoS trust, buffer policy, or test geometry changes, the affected tests should be rerun. NVIDIA's 5.18 release made dynamic ECN generally available ^[17], illustrating why release drift changes the tested system.

End-to-End Preflight and Configuration Freeze

Acceptance begins with a diffable, hop-by-hop matrix. NVIDIA's Ethernet Storage Fabrics guide says every network element in the path must handle RoCE ^[18]. Microsoft similarly describes PFC as a link mechanism between directly connected peers ^[19]. Data Center Bridging Exchange can expose conflicting NIC and link-partner QoS parameters ^[20], and carries its settings over LLDP ^[21]. One correct leaf does not compensate for one incorrect host or spine.

Table 1 is the configuration consistency matrix to complete for every tested path. Values shown as “design” must come from the approved low-level design, bill of materials, or statement of work.

The table is deliberately symmetric. A mismatch in either direction can produce an asymmetric stall. Juniper's QFX procedure separates RoCE data and CNP classifiers ^[22], while Cisco NX-OS 10.5(x) requires the PFC-priority and QoS-group mapping to agree ^[23].

Run these preflight checks before generating congestion:

1. Verify path and MTU: test both directions and every ECMP path available to the harness. Confirm host, routed interface, and switch-port MTUs with non-fragmenting probes and RoCE traffic. The upstream ip link set interface is the documented mechanism to change a host device MTU ^[24].

2. Verify classification: capture a packet at the source and at a reachable observation point. Confirm UDP 4791, DSCP, VLAN PCP where present, ECN-capable codepoint, and lack of unexpected remarking.

3. Verify host mapping: inspect application Type of Service, socket priority, user priority, hardware traffic class, and trust state. NVIDIA documents user-priority to traffic-class mapping through mlnx_qos or DCBX ^[25]. Its version-pinned interface can set trust to PCP or DSCP ^[26].

4. Verify switch mapping: inspect classifier, switch priority, queue, service pool, scheduler, ECN policy, PFC state, and CNP class on every element.

5. Verify persistence: compare running and startup configurations. NVUE changes take effect after apply ^[27], and saving creates the startup configuration file ^[28].

Lossless and Lossy Congestion-Control Modes

Lossless RoCE usually means ECN plus PFC on the selected priority, with reserved or dynamically managed lossless headroom. Lossy RoCE means ECN-based feedback without PFC. The ESF guide explicitly permits ECN-only lossy mode ^[29]. Cisco's blueprint says both hosts and network nodes must participate in the documented ECN and PFC design ^[30]. Neither label guarantees correct operation. Acceptance must test the chosen mode, not assume one is intrinsically healthy.

The signaling sequence under congestion is:

• Queue growth: an egress queue crosses its configured marking condition.

• ECN mark: the switch sets Congestion Experienced on an ECN-capable packet rather than necessarily dropping it. ECN uses a two-bit IP field with four codepoints ^[31].

• Notification: the destination NIC observes the mark and sends a CNP toward the original source. Cisco describes that receiver-generated feedback ^[32].

• Reaction: the source NIC's reaction point uses the CNP to reduce injection rate. A handled-CNP counter is therefore stronger evidence than a CNP merely arriving.

• PFC backstop: in a lossless design, the congested receiver can pause the selected priority on the adjacent link. PFC can propagate hop by hop, so direction and topology matter.

PFC is intentionally narrower than link-wide PAUSE. IEEE says it resembles 802.3x PAUSE but operates on individual priorities ^[33]. That selectivity still permits head-of-line blocking for unrelated flows sharing one queue. Peer-reviewed systems work identifies head-of-line blocking, congestion spreading, and occasional deadlocks as PFC limitations ^[34]. This is why more pause is not automatically better.

For a pinned example only, Cumulus Linux 5.18 documents switch priority 3 to traffic class 3 for RoCE and switch priority 6 to traffic class 6 for CNP ^[35]. It documents an absolute ECN threshold of 1,501,500 bytes for traffic class 3 ^[36]. Those values are evidence about that documented release, not recommended acceptance limits for another ASIC, NOS, port speed, or topology.

Baseline Counters and Observability

Capture a synchronized baseline after the fabric is quiet but before counters are cleared. Keep both the original snapshot and the post-test snapshot. This preserves evidence if clearing is unsupported or itself changes state. Hardware-offloaded RoCE may not appear in ordinary Ethernet counters ^[37], so collect RDMA and NIC hardware counters as well.

On Linux, rdma statistic show queries RDMA and driver-specific statistics ^[38]; with no object, it reports the default hardware counters ^[39]. Discover optional counters on the exact NIC and driver rather than assuming they exist.

Table 2 is a counter dictionary template. Replace every example name with the exact name and unit exposed by the installed build. “Project trigger” is deliberately not populated with a generic threshold.

The counter chain separates congestion from loss. Cumulus exposes a no-buffer discard for RoCE receive traffic ^[40]; Linux mlx5 describes a receive-buffer discard as a packet discarded because buffers were unavailable ^[41]. NVIDIA's NIC out_of_buffer can instead indicate lack of work queue entries ^[42]. These counters point to different remediation domains.

Counter semantics also affect timing. Linux RDMA hardware reads can be cached, with a documented default lifespan of 10 milliseconds ^[43]. Arista clears watchdog counters when PFC or its watchdog is disabled ^[44]. Record collection intervals, clear operations, reloads, wraps, and configuration changes in the evidence log.

Controlled Load Ladder

Use increasing complexity so the first anomalous step narrows the search. Keep path, duration, warm-up, CPU and GPU affinity, message geometry, and process placement explicit. Perftest endpoints require matching mode-specific options ^[45]. It can sweep message sizes from 2 bytes through 2^23 bytes ^[46], measure bidirectional bandwidth ^[47], and save a report as JSON ^[48]. Its maintainers state that the benchmarks do not emulate application traffic ^[49].

Table 3 defines the acceptance ladder and required raw artifacts. Durations, fan-in, and pass criteria are inputs, not defaults.

The ladder prevents a collective from concealing a basic mapping error. Published RoCE research notes that topology-aware collectives can avoid incast and balance traffic ^[50], so incast must be exercised separately. Conversely, NCCL Tests can span multiple processes, threads, and CUDA devices ^[51], which tests rank placement and collective behavior that perftest does not.

Operational rules for every step are:

• Synchronize clocks and snapshots: capture a common test ID and UTC timestamp on all hosts and switches.

• Warm up consistently: separate initialization from measured intervals and state the number of warm-up operations.

• Change one axis at a time: message size, QP count, fan-in, direction, or path, but not all simultaneously.

• Preserve raw results: save stdout, stderr, JSON where supported, counter snapshots, command lines, environment, and packet samples.

• Stop on physical errors: do not tune congestion control around new CRC, link, or symbol errors.

• Limit injection: execute failure or storm simulation only when approved, isolated, reversible, and paired with an abort condition. SONiC's own test plan verifies restoration after a simulated PFC storm ^[52].

Data Analysis and Evidence

The minimum useful analysis is a before and after delta joined to workload time. For each counter, compute delta = after - before, unless the device reports a gauge or watermark. A negative delta is not traffic reversal; it usually indicates a clear, wrap, reset, reload, or collection mistake that must be recorded. Juniper Apstra formalizes this idea as a time window over which interface-counter increments are measured ^[53].

Use these quantitative joins:

• Offered load: requested send rate, active senders, message bytes, QPs, and duration.

• Delivered work: transmitted and received RoCE bytes, completed operations, and collective correctness.

• Congestion feedback: egress ECN marks, receiver CE observations, CNPs sent, CNPs received, and CNPs handled.

• Backpressure: PFC Tx and Rx frames plus pause duration on the exact priority and adjacent links.

• Queue pressure: current use, maximum use, persistent watermark, and no-buffer discards.

• End performance: throughput, median and tail latency, completion time, and variance across repetitions.

The expected causal chain in an intentional congestion trial is offered load, queue growth, ECN mark, receiver CNP, sender handling, rate response. PFC may appear in a lossless design if the approved queue condition is reached, but it should be correlated hop by hop. Cisco describes PFC as traveling per hop from congestion toward the source ^[54]. Dell's Enterprise SONiC material exposes a command for PFC statistics across Ethernet interfaces ^[55], and its RoCE checks include persistent headroom watermarks ^[56].

Retain the complete achieved-bandwidth series, not only an aggregate. RFC 6349 recommends logged output across the full test duration ^[57].

BDP (Hypothetical Example)

Bandwidth-delay product (BDP) is a first-pass occupancy calculation, not a switch threshold. RFC 6349 defines BDP in bits as round-trip time multiplied by bottleneck bandwidth ^[58] and divides by eight for bytes ^[59]. For an illustrative 400 Gb/s bottleneck and measured 20 microsecond round trip:

400,000,000,000 bits/s x 0.000020 s / 8 = 1,000,000 bytes, or ~0.954 MiB.

That number excludes implementation margins, in-flight frame completion, MAC and PHY delay, cable distance, packet-size distribution, pipeline delay, port speed transitions, multiple ingress contributors, cell rounding, shared-pool policy, and vendor safety factors. NVIDIA notes that PFC headroom includes PHY and MAC delay ^[60]. Use BDP only to make assumptions explicit, then use the validated ASIC and NOS design for actual buffer settings.

Troubleshooting by Counter Deltas

Troubleshooting starts at the first broken link in the evidence chain. Avoid interpreting any single counter without path, priority, direction, and time.

• Throughput collapses, ECN marks rise, CNPs are handled, no discards: rate control is active. Compare achieved throughput and latency with the project band, then examine whether marking begins at the intended queue condition.

• ECN marks rise, destination CNP sent stays flat: verify ECN-capable bits, destination notification-point counters, NIC firmware, and CNP generation. The mark reached a queue, but feedback did not complete.

• CNP sent rises, source CNP received or handled stays flat: trace reverse-path classification, CNP traffic class, scheduling, and source reaction-point configuration.

• PFC Tx rises on a switch, adjacent NIC or switch PFC Rx stays flat: check physical adjacency, priority mapping, counter direction, telemetry interval, and whether the inspected interface is the actual path.

• NIC receive-pause rises: Linux mlx5 documentation interprets this as the network being unable to absorb traffic from that adapter ^[61]. Follow the path toward the congested egress.

• NIC transmit-pause rises: the adapter is signaling that it cannot absorb incoming traffic. Linux describes that state as the adapter being unable to absorb traffic ^[62]. Inspect receiver queues, work queue entries, host scheduling, and application posting behavior.

• No-buffer discard rises before meaningful ECN or CNP: suspect classification, an ECN policy that is absent or too late for the validated design, an unobserved microburst, or insufficient buffer. Do not simply increase thresholds.

• Pause appears on the wrong priority: inspect DSCP or PCP trust and rewrite boundaries. DSCP selects per-hop behavior at each node ^[63]; a mapping mismatch can put data or CNP into the wrong queue.

• Buffer maximum is high but current use drains: a transient burst occurred. Arista describes watermarks as exposing bursty, transient ingress use ^[64]. Correlate its timestamp with offered load and pause.

• Current buffer use or pause persists after load: stop further escalation, verify the generator ended, inspect watchdog and queue state, and follow the approved recovery plan.

• Only one direction fails: compare route, ECMP member, MTU, trust, CNP reverse path, and receiver resources independently. RFC 6349 recommends measuring directions separately before simultaneous bidirectional testing ^[65].

• Collective stalls but perftest passes: compare rank-to-NIC affinity, actual collective paths, message sizes, process placement, and control-plane connectivity. A point-to-point pass does not exercise the collective geometry.

This sequence separates a switch congestion condition from a receiver resource problem. Junos OS Evolved 24.4R1 exposes PFC and ECN monitoring through telemetry sensors ^[66]. Cisco NX-OS 10.4(x) can show real-time and peak shared-buffer state per port ^[67]. Use each platform's native view to reconstruct the same causal chain.

Acceptance Evidence Pack and Rollback

The acceptance pack is the product of the test. A dashboard screenshot is not sufficient because it cannot reproduce filters, intervals, resets, or raw values. Store immutable raw files plus a human-readable summary.

Required contents are:

• Scope record: purchase order or statement of work reference, design revision, test boundary, exclusions, and named acceptance criteria.

• Topology record: physical and logical diagrams, cabling, port maps, ECMP paths, rail mapping, and traffic-generator placement.

• Version record: switch model and ASIC, NOS and boot image, NIC part and PSID, firmware, driver, kernel, RDMA core, CUDA and collective versions.

• Configuration record: running and startup configs, QoS objects, trust and rewrite rules, PFC and ECN state, CNP class, queue scheduling, buffer policy, and checksums.

• Test manifest: command line, environment, source and destination, path, direction, message sizes, QPs, ranks, fan-in, duration, warm-up, repetitions, and offered load.

• Raw evidence: before and after counters, deltas, telemetry time series, packet captures, workload outputs, logs, and timestamps.

• Decision record: measured result, criterion, pass or fail, exception owner, remediation, retest reference, and sign-off.

• Rollback record: pre-test state, every temporary change, clear or reset action, restoration proof, and post-test quiet baseline.

Do not clear counters without recording it. Cumulus has an interface-specific RoCE clear action ^[68], but no portable host-side reset semantics were found across the examined stacks. The safe cross-platform method is synchronized snapshots and explicit deltas. Where counters wrap or reset, invalidate that interval and repeat it.

Rollback means more than removing the traffic generator. Confirm that temporary QoS changes are gone, test VLANs or routes are removed if required, watchdog or telemetry state is restored, buffers have drained, links are stable, and counters remain quiet for the project-defined observation window. GPU Smith describes integration and acceptance testing against written criteria ^[69]; as an adjacent advisor, it belongs in this evidence and decision process, not in a vendor feature comparison.

Implications and Future Directions

RoCE acceptance is moving from static configuration inspection toward closed-loop observability. A mature system should join switch queue telemetry, NIC congestion-control counters, workload timestamps, and topology in one time base. The immediate benefit is not a prettier dashboard. It is the ability to distinguish marking, feedback, backpressure, and loss in the order they occurred.

Three design implications follow:

• Telemetry is part of acceptance: if ECN marks, CNP reception and handling, pause direction, buffer maxima, and discards cannot be observed on the installed versions, the fabric is not yet supportable enough for production.

• Unused bandwidth may remain shareable: IEEE's Enhanced Transmission Selection allows other traffic classes to use available bandwidth ^[70]. Acceptance should therefore record background traffic as well as the RoCE class.

• Version control is part of performance: driver, firmware, NOS, QoS profile, and counter schema must be managed together. DOCA notes that its exact counter list depends on driver version ^[71].

• Thresholds are design outputs: link speed, topology, cable distance, MTU, packet-size distribution, ASIC buffer architecture, and endpoint response all matter. NVIDIA warns that an incorrect cable-length setting can waste buffer ^[72].

Dynamic and telemetry-assisted congestion controls will change exact commands and perhaps reduce reliance on PFC, but the acceptance logic remains durable: verify classification, create controlled pressure, observe the complete feedback loop, prove the absence or boundedness of loss and pause against supplied criteria, then preserve the evidence. ECN itself does not guarantee zero loss. RFC 3168 states that a full queue can still force a packet drop ^[73].

Frequently Asked Questions (FAQs)

What is the shortest useful RoCE validation checklist?

Freeze versions and topology, prove MTU and classification end to end, capture idle switch and NIC counters, run a single flow in each direction, add incast and bidirectional pressure, run the target collective, calculate deltas, compare them with buyer-supplied criteria, and restore the original state. The checklist is incomplete if it omits raw artifacts or CNP handling.

Should PFC pause counters be zero?

Not universally. In an ECN-only design, PFC should not be part of the data path. In a PFC-enabled design, pause may occur under the approved pressure case, but its priority, direction, duration, propagation, and post-load recovery must satisfy the project criteria. IEEE's objective is to prevent congestion-related frame loss ^[74], not to maximize pause traffic.

How is RoCE ECN validation different from CNP testing?

ECN validation proves that the intended queue marks ECN-capable packets under a controlled condition. CNP testing proves the destination recognizes those marks, sends feedback on the correct class, and the source receives and handles it. NVIDIA's rp_cnp_handled definition ties handling to transmission throttling ^[75].

What constitutes zero-loss evidence?

It is the absence of disallowed positive deltas in every relevant switch discard, NIC buffer-discard, host out-of-buffer, and physical-error counter for the exact test interval, combined with complete workload results. It is not a single switch counter at zero, and it does not prove all future workloads are lossless.

When should an acceptance team change ECN or PFC thresholds?

Only when the validated design identifies the platform, release, reason, target value, and rollback. A failed test should first identify classification, visibility, endpoint response, or resource problems. It should not trigger trial-and-error threshold changes. The current Cumulus guidance recommends leaving dynamic settings to the platform unless the operator understands the lossless requirements ^[76].

Conclusion

A defensible RoCE fabric acceptance result is a synchronized body of evidence, not a throughput screenshot. It begins with a frozen manifest, proves packet classification and feedback across every hop, observes idle state, and applies a load ladder that separates point-to-point transport, incast, bidirectional behavior, and collective traffic. The analysis then joins workload time to ECN, CNP, PFC, buffer, discard, and host-resource deltas.

The most important discipline is to preserve distinctions. An ECN mark is congestion signaling, a handled CNP is endpoint response, a PFC frame is one-hop backpressure, and a discard is loss. High buffer use is not itself a failure, while a zero snapshot is not itself proof. Direction, priority, timing, version, and topology give each counter meaning.

Finally, the test must use the acceptance limits supplied by the validated design or statement of work. BDP arithmetic can expose assumptions, and documented vendor defaults can explain one pinned release, but neither is a universal threshold. A fabric is ready for production when it meets the agreed performance and loss criteria, produces a coherent counter chain under controlled load, remains observable enough to diagnose deviations, and returns cleanly to its recorded baseline.