GPU Cluster Time Synchronization: NTP vs PTP Acceptance Matrix

Executive Summary

Choosing between the Network Time Protocol (NTP) and the Precision Time Protocol (PTP) for a GPU cluster is a per-role decision, not a cluster-wide one. The sources reviewed for this report as of October 2026 support a split design: chrony-class NTP for control-plane, management, storage and monitoring clocks, and a measured PTP domain only where a named trace, telemetry or application requirement cannot be met by NTP. The RFC for NTP version 4 describes typical clients on fast local networks as within a few hundred microseconds ^[1], and Red Hat documents tens of microseconds on a LAN ^[2]. Documented tolerances in common cluster software are far looser: Ceph Monitors warn at 0.05 seconds of drift ^[3], Kerberos defaults to 300 seconds ^[4], and MUNGE credentials live five minutes by default ^[5].

The concrete example that motivates this report is documented by NVIDIA. In NVIDIA System Management (NVSM) multinode mode, if compute node and aggregator time are not synchronized, compute node MQTT messages are assumed stale ^[6] and NVSM CLI commands error out ^[6]. The guide requires NTP-synced clocks ^[6] and states no numeric skew threshold, so the margin must be measured on the buyer's own aggregator and nodes. That is an acceptance test for the time source, not an argument for PTP.

PTP is justified by specific requirements. NVIDIA's Firefly reference design contrasts millisecond-level NTP ^[7] with applications that need tens of nanoseconds to microseconds ^[8], and it keeps control-plane nodes on NTP ^[8]. NVIDIA's Nsight Systems documentation gives NTP-based multi-report alignment errors of one to tens of milliseconds ^[9] and recommends PTP-class synchronization for detailed distributed analysis ^[9]. Neither source claims that every AI workload needs sub-microsecond time, and no source reviewed claims PTP improves GPU compute correctness.

Failover and holdover decide whether timestamps stay usable. Grandmaster datasheets quote 24-hour holdover drift from hundreds of microseconds for a standard crystal down to under one microsecond for rubidium ^[10], while PTP carries a timescale that differs from UTC by 37 seconds ^{[11] [12]}. The report therefore specifies an acceptance matrix for offset, drift, source loss, re-lock and log ordering, plus a worst-case relative timestamp bound equal to the sum of two absolute offset bounds. That bound is arithmetic, not measured accuracy. The decision rule: write down the maximum permitted skew from your own application and statement of work, choose the simplest protocol that meets it with margin, and prove it with evidence from normal operation and from a deliberate time-source failure.

Introduction and Background

This report is an analyst review of public documentation, not a benchmark. It contains no measurements taken by the publisher, and it makes no claims about any specific customer deployment. Every figure below is attributed to the document that states it, and where a document states no number, the report says so.

The audience is enterprise GPU cluster operators, network engineers, observability owners and air-gapped platform teams. Their shared question is practical: for each class of clock in a cluster (a head node, a baseboard management controller, a host operating system, a data processing unit, a switch), should the clock follow NTP or PTP, and what evidence shows it is good enough? "Good enough" is deliberately not defined by a universal threshold here. The buyer's application and statement of work (SOW) define the maximum permitted skew, and the purchased fabric defines what is physically achievable.

Three ideas anchor the analysis:

• Several clocks per server. NVIDIA's DGX service documentation describes a hardware real-time clock that the operator sets with hwclock -w after setting the date when NTP is not used ^[13].
• Different timescales. NTP's era 0 is defined in UTC from 1 January 1900 ^[1], whereas the PTP epoch is 1 January 1970 00:00:00 TAI ^[14].
• Path dependence. A protocol is only as good as its path: Red Hat advises PTP-enabled hardware between PTP clocks for best accuracy ^[15].

The report proceeds in order: clock-dependent failure modes, time sources and trust boundaries (connected and air-gapped), NTP and PTP domains by node role, evidence for offset, drift, holdover and failover, acceptance checks for NVSM and cross-layer correlation, and an alert, change and incident record.

Time-Domain Map: Which Clock Matters Where

The documented NVSM stale-message behaviour

NVSM multinode is a useful anchor because NVIDIA documents both the dependency and the symptom. The aggregator is the central coordinator for the compute nodes. The guide states why time matters: since the nodes and aggregator connect over MQTT, the timestamp on MQTT messages needs to be in sync ^[6]. It names the service, "ntpdate or chrony", as acceptable ^[6]. If synchronization is missing, the compute node MQTT messages are assumed stale ^[6].

Version and prerequisite facts, as of the 26.06 guide:

• Compute-node version: NVSM multinode supported version 26.03.01 onward ^[6], whereas the 25.03 guide stated 24.03.05 ^[16].
• BCM guidance: the Base Command Manager (BCM) provisioning page says multinode works on the older version but recommends 26.03.01 or above ^[17].
• Aggregator placement: on BCM-managed DGX SuperPOD deployments the aggregator runs on the BCM head node ^[17], so the head node's clock is the aggregator's clock.
• Telemetry: out-of-band telemetry is based on BMC Redfish URIs ^[6] and is stored with timestamps for historical analysis ^[6], so the aggregator is where host-clock and BMC-clock domains meet.

The guide gives no staleness window in seconds, so the tolerated skew is unknown until tested, and it describes NTP-class synchronization only.

Six clock domains and their trust boundaries

Table 1 below maps the clocks in a typical NVIDIA-based GPU cluster to a protocol choice, an upstream source, a failure behaviour to document and an owner. The protocol column is a recommendation derived from the cited documents; the "owner" column is a suggested accountability assignment, not a vendor statement.

Table 1 is a starting design, not a standard: NTP governs the management plane, and PTP, where used, is limited to the data-plane path with hardware timestamping from grandmaster to host. Mixed NTP-and-PTP is the likeliest source of hidden error, which is why Firefly and Cumulus both document that the two protocols must not both discipline the same system clock ^[22].

Host, DPU and BMC clock relationships

The relationships between clocks are documented only partially. In the Firefly reference architecture, a host-side virtual function shares the DPU's PTP hardware clock ^[8], and the PHC runs on the TAI timescale while system time is UTC, so the validation procedure expects a difference of 37 seconds ^[8]. By contrast, no NVIDIA document reviewed describes a PTP-disciplined BMC clock for DGX systems; the BMC clock evidence found is NTP-based and firmware-specific, and some of it concerns the older DGX A100. The relationship between a BMC clock and the host clock should therefore be treated as an acceptance-test item for the exact purchased platform, not an assumption.

NTP vs PTP for GPU Clusters: What Each Protocol Delivers

Accuracy claims, with their originators

The NTP specification gives modest typical numbers. RFC 5905 says typical primary servers on modern machines are precise within a few tens of microseconds ^[1], secondary servers and clients on fast LANs within a few hundred microseconds ^[1], and wide-area clients within a few tens of milliseconds ^[1]. Red Hat's RHEL 9 documentation states typical Internet accuracy of a few milliseconds ^[2] and LAN accuracy of tens of microseconds ^[2], adds that NTP hardware timestamping may reach sub-microsecond accuracy under ideal conditions ^[2], and says PTP relies on assistance in network switches and routers ^[2], while NTP hardware timestamping requires a polling interval shorter than the default range ^[2].

For PTP, the IEEE 1588-2019 abstract says the standard supports synchronization accuracy and precision in the sub-microsecond range with minimal network and computing resources ^[23], and NVIDIA's switch documentation repeats "sub-microsecond" as a capability ^[19]. None of these statements is a guarantee for a particular site; NVIDIA's Cumulus documentation warns that 1G links may have lower PTP accuracy because of hardware limitations ^[19].

NTP vs PTP for GPU clusters: the decision rule

Use the following decision rule per node role. Each criterion must be tied to a written requirement.

• NTP suffices when the strictest consumer is a management service, a scheduler or a log pipeline with a documented tolerance measured in tens of milliseconds or more. Slurm requires synchronized clocks across the cluster ^[24], and MUNGE needs the fastest and slowest clocks in the realm to fit within the credential time-to-live ^[5]. Ceph requires NTP or PTP daemons on Monitor hosts ^[25] and advises running NTP servers on bare metal, because virtualized clocks are unsuitable ^[3].
• PTP is justified when a named trace or application requirement needs finer relative ordering than NTP can evidence. Examples with sources: the Nsight Systems guide's Ethernet switch metric sampling requirements state that PTP is strongly recommended, especially when correlating samples from multiple systems ^[9].
• A third option exists when PTP is unavailable: the Nsight Systems vClock plugin measures relative offsets between system clocks and targets correlation error on the order of tens of microseconds ^[9].

The sources do not support the claim that all distributed training needs sub-microsecond time. They support a narrower statement: if your acceptance criteria include cross-node timeline correlation finer than the NTP error you can evidence, you need PTP, a vClock-style relative measurement, or both.

Chrony vs linuxptp for a GPU cluster

The two tools are complementary, not rivals. Chrony is an NTP implementation; linuxptp is a PTP stack whose ptp4l daemon implements boundary, ordinary and transparent clocks ^[26] and defaults to hardware timestamping ^[26] and to the end-to-end delay mechanism ^[26]. Hardware timestamping needs NIC support: chrony's hwtimestamp directive needs Linux 3.19 or newer and a NIC that supports hardware timestamping ^[27], and NIC capabilities are checked with ethtool -T ^[28]. The linuxptp helper phc2sys is typically used to synchronize the system clock to a PTP hardware clock ^[12], and Red Hat notes it is needed only with hardware time stamping ^[15].

Two integration points matter operationally. First, because the PHC follows the PTP timescale while the system clock follows UTC, phc2sys maintains the offset between them ^[12], and a wrong offset can leave the system clock wrong ^[12]. The ptp4l default for the TAI-UTC offset is 37 ^[26], matching the IERS statement that UTC-TAI has been -37 seconds since 1 January 2017 ^[11]. Second, chrony can consume a PHC as a reference clock, and its documentation says a PHC that runs on TAI (for example one disciplined by a PTP daemon) needs its UTC-TAI offset configured ^[27]. Record which daemon writes to the system clock, because two writers produce conflicting corrections.

Chrony's own tracking output provides the evidence for NTP nodes: "System time" is the current offset between the NTP clock and the system clock ^[29], and "Last offset" is the estimated local offset at the last clock update ^[29]. For ptp4l, summary statistics are reported in nanoseconds and parts per billion ^[26].

Leap seconds and UTC behaviour

NTP carries leap warnings in-band: a leap indicator value of 1 means the last minute of the day has 61 seconds ^[1]. PTP's timescale is continuous and shifted against UTC by tens of seconds ^[12]. IERS Bulletin C 72, dated 6 July 2026, announced that no leap second will be introduced at the end of December 2026 ^[11], and the 2022 General Conference on Weights and Measures resolution decided to increase the maximum UT1-UTC difference in or before 2035 ^[30]. Appliances still differ in timescale handling. Chrony offers a leapsecmode choice, in which the default system mode lets the kernel step the clock backwards when inserting a leap second ^[27], and for sources that do not announce leaps a timezone with leap data must be updated at least 12 hours ahead ^[27]. Amazon documents that its PTP hardware clock offers no smeared time option ^[31], so mixing smeared and non-smeared sources needs an explicit policy; its local hardware clock also requires ENA driver version 2.10.0 or later ^[32]. Leap-second behaviour of a specific grandmaster, switch or BMC must come from that appliance's documentation and be tested.

Precision Time Protocol in AI Data Center Infrastructure: NVIDIA Reference Design, Versions and Questions to Ask

Reference topology: PTP configuration for an AI training cluster

NVIDIA's reference design guide (RDG) for DOCA Platform Framework with Firefly builds a hierarchy: a grandmaster, an SN3700 switch acting as PTP boundary clock ^[8], and BlueField-3 DPUs running Firefly as PTP clients that synchronize the DPU hardware clocks. The host clock is then disciplined from the DPU. The profile aligns closely with the ITU-T G.8275.1 telecom profile ^[8], and the switch boundary clock is configured to meet G.8273.2 Class C requirements ^[8]. The Firefly service supports two trusted-function interfaces for PTP link redundancy ^[8], and the DPF Firefly reference includes a G.8275.1 profile configuration that users may adjust ^[33].

The reference design has stated limits that matter for procurement. Its grandmaster is a Linux server used for demonstration, and the RDG says it may not meet formal PTP grandmaster clock performance standards ^[8]. The RDG's validation monitor reports root mean square offset from the master in nanoseconds ^[8], but the page text reviewed gives no numeric pass threshold, so the acceptance threshold remains the buyer's.

Versions and prerequisites, as of October 2026

• Firefly guide (DOCA): firmware must be 24.34.1002 or higher ^[22], and supported BlueField image versions are 3.9.0 and higher ^[22]. Always check the version-specific guide for your DOCA release.
• DPF integration: the DPU flavor disables NTP on the DPU because Firefly needs a functional PTP setup ^[33].
• Cumulus Linux (documentation for 5.18): the documentation describes hardware timestamping of PTP packets ^[19], end-to-end delay only ^[19], and a single PTP domain per network ^[19].
• Onyx: NTP must be disabled before PTP is enabled ^[34], and PTP is documented as unsupported on Spectrum-3 based switches ^[34].
• NIC and DPU hardware: the ConnectX-7 datasheet lists G.8273.2 Class C compliance ^[35]. Time stamping is off by default and must be enabled before use ^[28].

A change-control caution applies to switches. Enabling the PTP service through the NVUE command line makes the platform restart switchd, which resets all network ports ^[19], so enabling PTP is a maintenance-window change, not a live toggle.

What Spectrum-X time synchronization does and does not tell you

The Spectrum-X question is common in search queries, so the evidence is stated plainly. The PTP documentation is generic to Spectrum switches running Cumulus Linux. NVIDIA NetQ monitors PTP on Spectrum switches running Cumulus Linux 5.0.0 and later ^[36]. No Spectrum-X-specific accuracy figure was found, and no per-ASIC accuracy table exists on the Cumulus page. The consequence for buyers is direct: whether hardware timestamping and boundary-clock support exist in the exact purchased fabric must be confirmed from the vendor's documentation for that switch model, operating system release and NIC firmware, then proven by measurement.

Hardware details that affect timestamps

Enabling time stamping on ConnectX-4 and later adapters disables receive completion-queue compression ^[28]. ConnectX-6 Dx and later devices default to free-running internal time mode ^[28], only physical functions can modify the hardware real-time clock ^[28], and all functions on one device must sync to the same 1588 master ^[28]. Each point bears on how a DPU-hosted or SR-IOV design is validated.

Source Failure, Holdover and Failover

Grandmaster choices for a GPU cluster

A grandmaster is the PTP time source for the domain. NVIDIA's RDG states that its demonstration grandmaster may not meet formal grandmaster performance standards ^[8]. Microchip's SyncServer S600 datasheet gives 24-hour holdover drift of 400 microseconds with a standard oscillator, 25 microseconds with OCXO and under 1 microsecond with rubidium, evaluated after five days locked to GPS ^[10]. Meinberg publishes an oscillator holdover table, and its LANTIME grandmasters state that the oscillator determines the holdover characteristics ^{[37] [38]}. Safran's VersaSync datasheet offers a miniature rubidium option and can be re-synchronized from an alternative external reference ^[39]. The Microchip S650 lists four gigabit ports with NTP and PTP hardware timestamping ^[40].

The Open Compute Project's Time Appliances Project describes the Time Card as GNSS-disciplined with a high-stability holdover oscillator to maintain accuracy if GNSS fails ^[41], and an Open Time Server Cluster workstream extends the Data Center PTP Profile to clusters of PTP grandmasters ^[42]. Meta describes a fleet design committed to supporting up to one million clients per appliance ^[43], distributing TAI to clients ^[43], and rehearsing holdover by disabling GNSS constellations on the Time Card receiver ^[43]; the transferable lesson is that failover was exercised deliberately. Meinberg lets operators configure the priority of every reference input ^[37] and delays falling back from GNSS until the internal oscillator can no longer beat the alternative ^[37], which matters when designing air-gapped fallback chains.

Best master clock behaviour and failover timing

Failover is decided by the best master clock algorithm (BMCA). In linuxptp the number of missed Announce messages before the last Announce expires is announceReceiptTimeout ^[44], default 3 ^[44], and the clock class threshold at which a client leaves lock state defaults to 248 ^[44]. The clockClass option denotes traceability of the time distributed by the grandmaster ^[26]. A G.8275.1 summary notes that a grandmaster in holdover within specification advertises a distinct clock class from one with no reference ^[45], and that the profile uses an alternate BMCA permitting multiple active grandmasters for redundancy ^[45]. For fixed topologies, ptp4l can skip the BMCA and use static roles ^[44]. At Meta, a time appliance that loses GNSS advertises a holdover clock class so that clients migrate away ^[43], and its design increases the window of uncertainty at a compound rate based on oscillator stability ^[43].

Cumulus supports only one PTP domain per network ^[19], so redundancy must live inside one domain, and failover time (announce timeout, re-selection and servo re-lock) must be measured with the actual switches and DPUs, not inferred from defaults.

Air-gapped sites and NTP fallback

Air-gapped sites have no public NTP pool or default internet reference. Cumulus Linux cannot learn NTP servers from DHCP ^[46], whereas NVOS enables NTP over DHCP by default. An air-gapped design must configure an internal reference explicitly, such as a GNSS-disciplined time server in a controlled location, or a rubidium-backed appliance that can be re-synchronized from an alternative reference ^[39]. If Network Time Security (NTS) is used internally, its key establishment runs over TCP port 4460 ^[47], which a firewall policy must permit.

Chrony offers several controlled-degradation mechanisms. The local directive with orphan is intended for isolated networks where hosts must agree with each other even if they are not synchronized to real time ^[48]; failover among orphan servers goes by reference ID order ^[48]. Raising minsources can improve reliability ^[48], maxdistance rejects sources beyond a default root distance of 3 seconds ^[27], and makestep is normally for boot only ^[27]. A tracking Reference ID that shows local mode means the host is not synchronised to any external source ^[29], which is the state the alert should detect. The older ntpd has its own orphan mode, enabled only after a delay with no sources ^[49]. An isolated cluster that agrees with itself is internally consistent but not traceable to UTC, and the record should say which condition applies.

Drift between corrections

When all references fail, clocks drift at the oscillator's frequency error. Chrony's maxclockerror assumes 1 ppm by default ^[48]. At 1 ppm, a 50 millisecond relative error (the Ceph monitor limit ^[3]) accrues in about 14 hours, an arithmetic illustration and not a measurement of any server. A real server's rate error must be read from chronyc tracking, where Frequency is the rate error and Skew is the estimated error bound on it ^[29].

Data Analysis and Evidence

Documented tolerances and capabilities

Table 2 collects numbers that bound the requirement from above (what software tolerates) and from below (what each protocol or device claims). The entries are vendor or project documentation for each product's own facts. Several are defaults or typical values, not guarantees.

Two observations follow. None of the cluster management tolerances approaches the sub-microsecond range, which supports NTP for management. The only entry that argues for PTP is the trace-alignment one: Nsight Systems' NTP-based error range of one to tens of milliseconds is wide enough that sub-millisecond ordering between nodes cannot be asserted from NTP alone.

The relative timestamp bound

Two events logged on different clocks can be ordered reliably only if their true separation exceeds the combined uncertainty of both timestamps. The simple arithmetic bound is:

relative timestamp uncertainty bound = |offset bound of clock A| + |offset bound of clock B|

This is an arithmetic bound derived from two assumed absolute offset bounds. It is not measured system accuracy, and it ignores timestamping point error (an application logging after a scheduling delay), queueing and clock steps.

(Hypothetical Example) If an audit of the NTP clients in a rack bounds clock A at 2 milliseconds and clock B at 3 milliseconds, the bound is 5 milliseconds, so two log lines closer than 5 milliseconds apart cannot be ordered with confidence. If the PTP-disciplined hosts' audited bounds were 1 microsecond each, the bound would be 2 microseconds. Both pairs of inputs are invented for the arithmetic and are not measurements.

Chrony publishes a related quantity: the clock error bound, built from system time offset, root dispersion and half the root delay. Amazon notes that with a direct PHC connection chrony will underestimate this bound, so Amazon adds the PHC's own error bound ^[50]. The lesson for cluster operators is that a daemon's self-reported error is a lower estimate of the true uncertainty whenever another clock hop sits beneath it, such as a DPU hardware clock feeding a host.

Implementation: Evidence, Acceptance Tests and Records

Collect offset, drift, holdover and failover evidence

Evidence has to cover normal operation and a time-source failure. The following inventory is the minimum, using only commands and fields the cited tools document.

• NTP nodes: record chronyc tracking System time and Last offset ^[29], source reachability, and whether the reference ID shows local mode ^[29].
• PTP hosts: record the ptp4l summary offset RMS and maximum offset, whose units are nanoseconds and parts per billion ^[26], and the phc2sys offset between the PHC and CLOCK_REALTIME. Confirm the TAI-UTC offset matches the expected 37 seconds.
• Switches: collect the Cumulus PTP monitor counters and the last 25 PTP timestamps from the timestamp log ^[19], and use NetQ for fleet-level PTP views.
• NICs and DPUs: record ethtool -T output and the firmware and image versions against the Firefly prerequisites.
• Observability pipeline: record each exporter's sampling and scrape behaviour, so a sample's timestamp is not assumed to be the scrape instant.

No document reviewed prescribes a minimum collection window; the buyer must set one that spans normal load and a maintenance event, and record the reason.

GPU cluster clock synchronization acceptance testing: offset, drift, source loss, re-lock and log ordering

Table 3 is an acceptance matrix. The pass criteria are intentionally left as references to the buyer's own requirement, because no universal offset threshold exists in the sources and none should be invented. Vendor defaults appear only as reference points.

Table 3 deliberately includes a test that offsets a lab node's clock to find the NVSM staleness margin. Because the NVSM guide states no threshold, this test converts an unknown into a number. It should run only in a lab or a maintenance window, never as a live experiment on a production aggregator.

Clock drift in distributed training logs: cross-layer correlation checks

Correlation fails quietly. The NVIDIA observability guidance lists a common timeline for GPU, node and fabric signals in one view as a selection criterion ^[51], and it names DCGM and NVSM Prometheus endpoints among the sources ^[51]. The page states no timestamp rule, so the common timeline depends on the clock work in this report.

Run a marker test: emit a uniquely tagged event at a known moment on a host, and confirm the same tag appears with the expected ordering in the host log, the BMC event log, the switch log and the application trace. Then evaluate the downstream pipelines:

• Metrics: Prometheus accepts out-of-order samples only within a configured window ^[52].
• Logs: Loki by default accepts entries only within a window of one hour behind the newest entry in a stream ^[53], and lists clock skew between log sources and Loki as something to check when ingestion errors occur ^[53].
• Traces: Jaeger's query service can adjust spans when a server span appears to start earlier than the client span ^[54], which hides skew from the viewer instead of removing it.
• Certificates: X.509 validity is an interval, so a badly wrong node clock also breaks certificate checks ^[55].

Operational alert, change and incident record template

The template below is a suggested structure. Bracketed items are fields to fill, not data.

TIME-SOURCE DRILL RECORD
Cluster / site: [name]            Date and window: [start - end]
Requirement source: [SOW clause / application doc]   Max permitted skew: [value, unit]
Roles under test: [control plane | BMC | host | DPU | switch | grandmaster]
Protocol and daemon versions: [chrony / linuxptp / Firefly / NOS / firmware]
Failure injected: [source removed | GNSS lost | link cut] at [time]
Alert fired at: [time]     Alert rule: [name]
Failover target: [source]  Failover completed at: [time]
Offset before / during / after: [bound] / [bound] / [bound]
Clock steps observed: [yes/no, where]
NVSM command state during event: [ok / error text]
Log ordering marker result: [pass/fail, pairwise bound]
Pass/fail vs requirement: [ ]     Owner sign-off: [name, role]
Follow-up changes: [ticket IDs]

Pair the drill record with an alert set:

• Source loss: loss of every external source, or a tracking Reference ID in local mode.
• PTP state: a port state change or a grandmaster identity change.
• Offset: any offset beyond the buyer's bound.
• Clock steps: any system-clock step.

Pair it with a change rule: switch PTP enablement is a disruptive change, and any change to which daemon writes the system clock needs a rollback plan because both Firefly and Cumulus document that NTP and PTP must not both set the clock.

Implications and Future Directions

Three implications follow for operators planning purchases and acceptance in 2026 and 2027.

• Time becomes a bill-of-materials line. Hardware timestamping and boundary-clock support are model-and-release questions: Cumulus documents boundary clock only, Onyx documents a Spectrum-3 limitation, and NIC behaviour depends on firmware and mode. Put these in the request for proposal beside the buyer's maximum permitted skew.
• Monitoring stacks raise the cost of bad clocks. NVIDIA publishes no skew threshold for the stale-message behaviour, so each operator should own a measured margin and re-measure after NVSM upgrades, since the minimum compute-node version moved from 24.03.05 to 26.03.01 between guide releases.
• Leap-second policy is converging but unfinished. With no leap second announced for the end of December 2026 ^[11] and an international resolution to widen UT1-UTC tolerance by 2035 ^[30], the operational risk is declining, but appliances still differ in timescale handling. Keep the A8 test in the acceptance matrix.

Open questions remain: no public benchmark in the sources measures achieved PTP offset on a Spectrum-X fabric end to end, and no NVIDIA document reviewed describes PTP for the DGX BMC. Operators should treat these as lines in their own test plans.

Frequently Asked Questions (FAQs)

Is NTP enough for a GPU cluster?

For management-plane and most telemetry use, the documented tolerances (0.05 seconds for Ceph ^[3], 300 seconds for Kerberos ^[4]) are far looser than typical LAN NTP accuracy of a few hundred microseconds ^[1]. NTP is enough unless a trace or application requirement exceeds what you can evidence with it.

When is PTP justified?

When a written requirement needs cross-node ordering finer than NTP can evidence, for example multi-system trace correlation, where NVIDIA's profiler documentation recommends PTP. It is not justified by the claim that AI workloads in general need it, and the sources reviewed do not say PTP changes GPU compute correctness.

What does a PTP grandmaster for a GPU cluster need?

A GNSS-disciplined or atomic-referenced appliance with holdover figures that match your outage budget (compare datasheet holdover tables ^{[10] [38]}), PTP profile support that matches the fabric, and a documented air-gapped fallback ^[39].

How do I prove clocks stay usable during a failover?

Run drills A3 to A5 in Table 3, keep the drill record, and compare observed offsets (chrony ^[29], ptp4l ^[26]) with the buyer's requirement, not with a vendor default.

Conclusion

The evidence supports a role-based answer to the question of GPU cluster time synchronization. Keep the control plane, BMCs, storage and monitoring on NTP with at least three independent references ^[56], confirmed by the A3 source-loss drill, and a documented air-gapped fallback. Reserve PTP for the data-plane path where a named requirement, such as multi-node trace alignment, justifies the added hardware, configuration and monitoring burden, and only after confirming that the purchased switches, NICs and DPUs provide hardware timestamping and boundary-clock support. NVIDIA's NVSM guide shows why even the NTP tier needs acceptance testing: documented stale-message behaviour with no published threshold.

The practical deliverables are an acceptance matrix with buyer-defined criteria, a source-loss drill with retained evidence, and an incident template that records the offset bound and any clock step. The worst-case relative timestamp bound (the sum of two absolute offset bounds) is a planning aid, not a measurement. Until those measurements exist for a given fabric, claims about achieved precision remain unknown, and an operator who has not measured them should not assert them.