NVIDIA Topograph Kubernetes Slurm: Private GPU Cluster Pilot

Executive Summary

NVIDIA Topograph is an open-source service that discovers a cluster's network topology from a provider (a cloud API or an on-premises fabric tool) and translates it for a scheduler through an engine, publishing Kubernetes node labels, Slurm topology files ^[1], or Slinky ConfigMaps ^{[2] [3]}. It does not place jobs. Engines "only translate" and never discover topology themselves ^[3], so every placement gain depends on a separate scheduler honoring the output. This report, current as of November 2026, gives private GPU cluster operators an adoption gate for deciding whether to pilot it.

The release picture needs pinning. NVIDIA's developer blog is dated 22 September 2026 ^[4], while the repository's latest GitHub release, v1.1.0, was published on 26 October 2026 ^[5] and added Crusoe and Firebird providers ^[5]. The blog's provider sentence names Google Cloud, Lambda, Nebius, Nscale and OCI, although its provider support table also lists Crusoe, so Crusoe support was already described before v1.1.0 shipped the provider code ^[4], whereas the README lists AWS, Crusoe, GCP, OCI, Nebius, Nscale, Lambda and NetQ ^[3]. Treat the repository tag, not the blog, as the support list.

Three distinctions drive the decision. First, node-local alignment (NUMA, PCIe, NVLink inside a server) belongs to the kubelet Topology Manager and the device stack, and Topograph is described as complementary to it ^[6]. Second, the default Kubernetes scheduler treats Topograph labels as a preference scored one pod at a time ^[6], whereas KAI Scheduler offers a hard required level ^[7] and Kueue offers required and preferred annotations ^[8]. Third, non-MNNVL (Multi-Node NVLink) InfiniBand clusters depend on Topograph with an InfiniBand provider as the only network topology source, because the GPU clique label is absent ^[9].

Refresh is event-driven. Labels refresh when generation runs after a watched change ^[4], changes to node metadata do not trigger the observer ^[10], and the Slurm trigger fires only on node up and down events ^[11]. A healthy pod also does not prove current topology ^[12]. The pilot therefore needs a wiring-change acceptance test, not just an install check.

Published evidence is mixed and not specific to Topograph. A production study reports a 10.6% end-to-end gain from topology-aligned scheduling on more than 9,600 GPUs ^[13], while an older fat-tree study found minimal network-placement impact on a tapered cluster ^[14]. No source reviewed quantifies Topograph itself, so any throughput or cost claim requires the buyer's own paired runs.

Introduction and Background

Large distributed training and inference jobs spend a measurable share of their time in collective communication, and where the ranks land in the fabric changes which links carry that traffic. NVIDIA's documentation states the premise plainly: nodes connected to the same leaf switch can communicate with lower latency and higher bandwidth ^[15]. Slurm has long supported topology-aware allocation from a configuration file, but the maintainers of Topograph argue that node churn makes a static file unreliable ^[16]. Their SC25 workshop abstract states that multi-node workloads should be scheduled on network-adjacent nodes ^[16].

This report examines how a GPU cluster network topology discovery service for Kubernetes and Slurm behaves in a private deployment: what Topograph does, what it explicitly does not do, and how to run an auditable pilot. It answers the search queries that bring operators to the tool: how Topograph works with Kubernetes and Slurm, how it relates to topology.conf, how it behaves on bare-metal InfiniBand, which node labels it writes, and how topology-aware scheduling differs between the default Kubernetes scheduler and gang-aware schedulers.

Two terms recur. A provider is the source of topology (a cloud API, NetQ, InfiniBand tooling), and an engine is the target format (Kubernetes, Slurm, Slinky, Node Feature Discovery, or a JSON graph). Discovery is asynchronous: a request is submitted, validated, and polled for a result.

This report is built on dated public specifications. It contains no benchmark runs, client results or partnership claims. Where the evidence is vendor-authored, it is labeled as such. Readers who act on commands below should validate them against the chart and documentation version they pin, and should involve their network and security engineers before granting any permissions.

Key Changes: The September 2026 Release and What Followed

What was published on 22 September 2026

NVIDIA's developer blog, published on 22 September 2026, presents Topograph as a bridge from provider-reported topology to scheduler-specific formats. The repository README frames the same role: Topograph sits between topology sources and schedulers and replaces neither. The latest GitHub release, v1.1.0, is dated 26 October 2026 ^[5]. The earlier v1.0.0 release was published on 18 August 2026, before the blog ^[5].

The practical consequence is that a pilot started from the blog alone may not match a pilot started from the current tag. Record the exact release tag, chart version and image digest in the pilot file. The README clones from github.com/dsx-ai-factory/topograph, whereas the release API and raw documentation links cited here use NVIDIA/topograph, so record the precise path you install from and confirm that it matches the release you pinned.

Provider inventory: three sources, three lists

The blog's prose names the cloud providers with working integrations as Google Cloud, Lambda, Nebius, Nscale and OCI, as noted above, but its support table also lists Crusoe, which the footnote says reads labels from Crusoe Managed Kubernetes nodes. The README names AWS, Crusoe, GCP, OCI, Nebius, Nscale, Lambda and NetQ ^[3]. The main-branch overview additionally describes a provider for sites that inventory fabric cabling in Nautobot. The v1.1.0 release notes list the addition of the Crusoe provider ^[5] and a Firebird provider with simulation ^[5].

These lists are not contradictions so much as snapshots taken on different dates. The engineering response is to build the provider decision from the pinned tag's documentation, and to treat the blog as a dated introduction.

A label contract that Topograph owns

On Kubernetes, the documented default keys are fabric.topograph.run/tier-N and accelerator.topograph.run/domain, with an optional sub-domain key. Tier 0 is closest to the compute node and tier numbers increase with distance ^[6]. Label values of 63 characters or fewer are used as-is, and longer values are hashed, which preserves locality but not the original identifier ^[9]. Not every provider produces both topology types.

One point corrects a common assumption. Topograph does not write the GPU Operator's clique label. There is no default accelerator-domain source label, and an operator who wants the clique label to be authoritative must set it explicitly.

Event-driven refresh, not polling

The Node Observer is a controller that monitors configured node and pod changes. Events include node adds and deletes and trigger-pod readiness transitions ^[12]. Bursts are coalesced, so with the default 15 second delay a result is never available sooner than 15 seconds after node churn stops ^[12]. NVIDIA's own caveat is that whether a fabric change becomes visible depends on the provider and its triggering events. The "Change detection, stale labels, rollback and owners" checklist below converts these statements into an acceptance test.

Provider-to-Scheduler Decision Matrix

Choosing a provider for a private cluster

Private clusters are usually one of four shapes: InfiniBand without NetQ, Spectrum-X or InfiniBand with NetQ, a multi-node NVLink (MNNVL) rack system, or a colocation or cloud lease where the vendor already exposes topology. NVIDIA's InfiniBand provider documentation recommends NetQ over direct discovery where NetQ is deployed, calling it the standard approach for Spectrum-X ^[17].

For the InfiniBand bare-metal variant, the provider discovers the switch tree using ibnetdiscover ^[18], is intended for bare-metal clusters such as Slurm, needs pdsh on the node running Topograph ^[18], and requires no cloud credentials. All InfiniBand variants are single-region only ^[17]. The Kubernetes variant is different in a way that matters to security reviewers: deployments typically run the broker privileged ^[18].

The NetQ provider queries the NetQ API for fabric topology and NVLink domain data ^[19], requires a running NetQ server reachable from the Topograph host, and treats NVLink domain discovery as best-effort, returning only the switch tree if that fails.

Table 1 below maps provider families to their topology source, whether they see cross-node fabric or only an NVLink partition, and the scheduler outputs that apply. It summarizes documentation for the v1.1.0 documentation set and should be re-derived from the tag you pin.

Two lessons come out of Table 1. The dra provider is not a general Kubernetes MNNVL topology provider: it does not discover the switch fabric between partitions, so it cannot guide placement when a workload spans more than one NVLink partition. And for the common private case of InfiniBand without MNNVL, NVIDIA states that Topograph with an InfiniBand provider is the only source of network topology for scheduling decisions ^[9].

Mapping provider output to scheduler artifacts

The engines translate one canonical graph into different artifacts. The decision is mostly about which artifact your scheduler can consume.

• Kubernetes engine: writes node labels directly through the Kubernetes API. It lists nodes once and patches only changed labels. This is the engine for native pod affinity, KAI Scheduler and Kueue.
• Node Feature Discovery (NFD) engine: publishes NodeFeatureGroup resources instead of labels. Use it only when a downstream component already consumes NFD. The NodeFeatureGroup API is alpha and must be explicitly enabled ^[23], and a scheduler still has to decide which leaf group to use.
• Slurm engine: produces tree or block topology through a plugin parameter. When block sizes are not configured, Topograph derives them from the discovered accelerator domains. If no output path is set, the text is returned in the HTTP response.
• Slinky engine: updates Slurm topology configuration stored in Kubernetes ConfigMaps. Slinky also reads a node annotation to place NodeSet pods in the topology ^[24], and a topology file must still exist for dynamic topology to work ^[24].
• Graph engine: returns per-instance network layers and accelerator labels as JSON, which is useful as the "discovered" side of the audit worksheet below.

Node-Local Affinity Versus Cross-Node Fabric Placement

Node-local: GPU, CPU and NIC proximity

Inside a server, the question is which GPUs, CPU cores and NICs share a PCIe switch, a NUMA node or an NVLink mesh. nvidia-smi topo -m prints that matrix. Its legend distinguishes NVLink connections ^[25], a single PCIe bridge, and paths that cross the inter-socket interconnect. Another mode shows CPU and memory affinity and the NUMA node for each GPU.

In Kubernetes, the Topology Manager is a kubelet component that coordinates resource alignment on one node ^[26]. Its default policy performs no topology alignment ^[26], while the restricted policy rejects a pod whose alignment is not preferred ^[26]. The feature has been stable since Kubernetes v1.27 ^[26]. NCCL, the collective library, detects intra-host topology itself, and its topology file must describe a single host ^[27].

Topograph's documentation contrasts the two layers directly: Topograph labels tell the scheduler which nodes belong together on the network, and both Topograph and the Topology Manager can be active simultaneously. A pilot should therefore confirm the node-local policy separately, because a correct fabric label cannot repair a pod whose GPU and NIC straddle NUMA nodes.

Cross-node: leaf, spine and NVLink domain

Cross-node placement concerns which servers share a leaf switch, which share a spine, and which share an NVLink domain. On GB200 NVL72 systems, NVIDIA describes 72 GPUs in a single NVLink domain with 130 TB/s of GPU communication ^[28]. The Kubernetes DRA (Dynamic Resource Allocation) driver for NVIDIA GPUs provisions ephemeral multi-node NVLink fabrics using IMEX ^[29] and requires Kubernetes 1.32 or later ^[29].

The clique label that identifies an MNNVL domain is written by NVIDIA's GPU stack, not by Topograph, and is scoped to nodes inside such a domain. NVIDIA's Topograph reference states that the label cannot tell a scheduler which racks share a top-of-rack switch ^[9], so rack and leaf placement still needs fabric tier labels. The label is also refreshed on an interval rather than propagated instantly ^[9]. On hardware where the GPU fabric never reports completion, the clique label is absent and the InfiniBand provider becomes the only source.

The separation matters for acceptance testing. A job can pass a node-local test (every GPU paired with the nearest NIC) and fail a fabric test (its eight nodes sit under four different leaves), and the reverse is equally possible.

Choosing the Scheduler Layer

Default Kubernetes scheduler: a preference, one pod at a time

With Topograph labels, native pod affinity can express "prefer the same leaf". Required affinity rules are hard filters, since the Pod is not scheduled unless the rule is met ^[30], whereas preferred rules are scored ^[30]. Topology spread constraints have the same split: DoNotSchedule is hard ^[31] and ScheduleAnyway is soft ^[31]. Topograph's Kubernetes engine page notes that the default scheduler places one pod at a time, so placement depends on where earlier pods landed ^[6], and it states that there is currently no fully network-aware scheduler.

Kubernetes is adding native alternatives. The standard scheduler evaluates Pods sequentially ^[32], but gang scheduling, in which none of the Pods are bound unless the group fits ^[33], is documented as beta since Kubernetes v1.37 and disabled by default ^[33]. Native topology-aware scheduling is documented as alpha since v1.36 and disabled by default ^[34]. It enforces a topology constraint strictly ^[34], allows a single constraint per PodGroup ^[34], and does not trigger preemption ^[34]. Topograph's own label reference describes the same KEP as alpha in v1.37 ^[35], a version discrepancy between two NVIDIA-adjacent pages that a pilot should resolve against the cluster's actual Kubernetes release.

KAI Scheduler and Kueue: gang placement with levels

KAI Scheduler derives topology from node labels and its own Topology custom resource ^[7], which carries an alpha API version. A required placement will not place the workload on nodes lacking the specified label ^[7], and a preferred placement falls back to a higher-level domain. Its multi-level documentation ties PodGroup gang semantics to these constraints. NVIDIA Run:ai documentation says users may apply labels manually or use discovery tools, and that labels are applied at the Kubernetes level and managed outside Run:ai ^[36].

Kueue topology-aware scheduling has been beta since Kueue v0.14 ^[8]. It assumes node labels are set by the cloud provider or by administrators of on-premises clusters ^[8], which is where Topograph's Kubernetes engine fits. A required annotation forces all pods into one domain at the named level ^[8], while the preferred annotation is a preference rather than a requirement ^[8] and escalates to the next level up when the PodSet does not fit ^[8].

Slurm: tree, block and per-partition topology

Slurm's default topology plugin is flat ^[37], so topology-aware behavior is something you turn on. Tree topology describes leaf switches with their nodes and recommends one top-level switch ^[38]; users cap leaf switches with --switches ^{[39] [38]}. Block topology, intended for NVLink domains, defines enforceable block sizes ^[38], each a power of two larger than the prior ^[40], and jobs use --segment ^[39]. An NVIDIA presentation at SchedMD's SLUG24 states that the block plugin was introduced in Slurm 23.11 ^[41]. Slurm 25.05 introduced multiple topologies selectable per partition ^[38], and nodes can be dynamically added to and removed from topologies ^[38].

Slurm itself cautions that best-fit allocation on currently free nodes can use more switches than the optimum ^[38]. File changes take effect upon restart, signal or reconfigure ^[40], which is why the Topograph Slurm path registers a trigger that runs whenever a node goes down or comes up ^[11].

Volcano as a different architecture

Volcano is a separate scheduler with its own model and should not be treated as equivalent. Its HyperNode resource represents a network performance domain, typically mapped to a switch (Source: volcano.sh). It supports a hard mode that keeps tasks in one HyperNode (Source: volcano.sh) and a soft mode that does so as much as possible (Source: volcano.sh). It can discover HyperNodes from data sources itself (Source: volcano.sh), polling at a default of one hour (Source: volcano.sh). Operators who already run Volcano should compare its discovery to Topograph's rather than stack both.

Table 2 summarizes the scheduler layer decision.

The table supports a plain rule: if jobs must start together inside one fabric domain, a preference in the default scheduler is not enough, and the choice is between a gang-aware layer (KAI, Kueue, or native PodGroup scheduling once enabled) and Slurm. If jobs tolerate fragmentation but benefit from proximity, a preferred affinity is the lowest-risk first step.

Implementation Considerations and Process Changes

Pin prerequisites and verify permissions before running commands

The README states that Kubernetes installs need Kubernetes 1.27 or later and Helm 3.10+ or 4.x ^[3]. The documented install uses the NGC Helm repository ^[42] followed by a helm install of the topograph/topograph chart and a helm test hook. A minimal sequence, to be checked against the chart README of your pinned version, is:

helm repo add topograph https://helm.ngc.nvidia.com/nvidia/topograph
helm install topograph topograph/topograph \
  --namespace topograph --create-namespace \
  --set provider.name=<provider> --set engine.name=k8s
helm test topograph --namespace topograph

Security review should precede the install because the service's design places controls on the operator. The API server does not implement built-in authentication ^[6], so access control belongs at the network layer. A NetworkPolicy has no effect unless the container network interface enforces it ^[6]. The node data broker has cluster-wide node update permission in the chart's RBAC, and an optional admission policy validates that the node being modified matches the node name claim ^[6]. Cloud-hosted deployments should treat instance metadata access as a credential-exposure vector. On large first reconciliations, the Kubernetes client defaults to 5 queries per second with a burst of 10 ^[6], which the chart exposes as tunables.

Cloud providers add their own read permissions: the AWS provider uses the EC2 instance topology API, the GCP provider the Compute Engine instance list, and Nebius its instance list. Grant the narrowest role the pinned provider page documents.

Stage one: simulation without production hardware

Model files can simulate discovered topology without querying a real cloud API or NetQ instance ^[43]. A helper script creates or reuses a local kind cluster, installs KWOK, and applies the generated virtual nodes. The test provider simulates the API lifecycle itself, and the infiniband-sim variant replays saved ibnetdiscover output with no hardware or credentials. The README points to a no-cloud demo script.

Use simulation to prove three things before touching production: that your scheduler configuration consumes the label keys, that required constraints reject an oversized gang cleanly, and that removing a node from the model removes stale labels.

Stage two: controlled production pilot

A controlled pilot uses a labeled subset of nodes, one scheduler path at a time, and no change to the existing production queue. The sequence below is the minimum.

1. Capture desired topology. From the network team's cabling plan, list each node's leaf, spine and (for MNNVL) rack domain.
2. Capture discovered topology. Record Topograph output (labels, topology.conf, or Graph engine JSON) and independent raw evidence such as ibnetdiscover output.
3. Diff. Any node whose discovered leaf differs from desired is a finding, regardless of which side is wrong.
4. Verify node-local policy. Record nvidia-smi topo -m and the kubelet Topology Manager policy for every pilot node.
5. Run paired jobs. Run the same job through the baseline and the topology-aware path, alternating order, on the same node pool.
6. Run the change test described in the next section.

For fabric measurement, nccl-tests reports bus bandwidth, which is meant to reflect the speed of the hardware bottleneck ^[44] and can be compared with hardware peak independently of rank count ^[44]. A multi-node build requires MPI, and the project's example launches 64 ranks over 8 nodes ^[45]. Discard warmup iterations, which are not timed ^[45].

Table 3 is a pilot run record template. The cells are placeholders to be filled from your own runs; the report supplies no measured values.

The interpretive rule for Table 3 is to compare rows only when the job, GPU count and node pool are identical, and to repeat each row enough times to show run-to-run variance before attributing any difference to placement. A difference smaller than the variance is not evidence of a gain.

Change detection, stale labels, rollback and owners

Treat the following as an acceptance checklist, each item tested by an actual change in a maintenance window or in simulation.

• Node added or removed. Expected to trigger regeneration. Verify new labels appear after the aggregation delay.
• Node metadata changed. Does not trigger the observer ^[10]; verify you issue a manual request, and note that results are kept only for the last 100 request IDs ^[12].
• Switch re-cabled with no node event. The Slurm trigger does not detect arbitrary rewiring ^[4]. Verify a scheduled or manual regeneration closes this gap, and compare the result with fresh ibnetdiscover output.
• Selectors match nothing. The topology never regenerates ^[12]. Verify selectors against real node labels.
• Continuous churn. The queue can starve ^[12]. Verify behavior under an autoscaling or repair storm.
• Provider failure. The server makes up to five generation attempts ^[12], so failures may not appear where you expect. Alert on API errors, not only on pod health, because the health endpoint does not test provider reachability, credentials or the last generation result ^[12].
• Metrics hygiene. The topograph_missing_topology gauge is only ever set, never cleared, so a nonzero reading may be stale.
• MNNVL clique lag. The clique label is refreshed on an interval; verify scheduling tolerates the lag.

Topograph states that its output reflects reported rather than intended topology ^[4]. A mis-cabled but self-consistent fabric will therefore yield labels that match the wiring and disagree with the design, which is why the desired-versus-discovered diff is a separate step.

Rollback is the operator's design, because the sources reviewed document no rollback procedure. It is an inference, not a documented guarantee, that labels are inert until a scheduler rule references them, so the lowest-risk rollback is to remove or relax the scheduler rule (required to preferred, then to none) before removing the service. For Slurm, keep the previous topology.conf and reload it, since file changes take effect on restart or reconfigure.

Define exit criteria before the pilot begins, so that the result is not argued after the fact:

• Label accuracy: zero unexplained differences between desired and discovered topology on the pilot nodes.
• Change coverage: every change type in the checklist either triggers a refresh within the agreed window or has a documented manual step.
• Performance: paired-run differences exceed the measured run-to-run variance, or the pilot is recorded as inconclusive.
• Safety: the rollback step has been exercised at least once.

Assign owners explicitly:

• Network or fabric team: owns the desired topology, cabling changes and the change calendar.
• Platform team: owns the Topograph release, chart values, observer selectors and alerts.
• Scheduler team: owns KAI, Kueue, Slurm configuration and the required versus preferred policy.
• Security team: owns RBAC review, network exposure and provider credentials.
• Workload owners: own the paired-run job definitions and the acceptance sign-off.

Data Analysis and Evidence

Hardware and latency numbers that frame the problem

Per-GPU NVLink bandwidth is 900 GB/s on the fourth generation in H100 ^[46] and 1.8 TB/s on the fifth generation in GB200 NVL72 ^[28]. Network links are slower by an order of magnitude: Quantum-2 InfiniBand switches provide 64 ports of 400 Gb/s ^[47], Quantum-X800 provides 144 ports of 800 Gb/s ^[48], and ConnectX-8 supports up to 800 Gb/s ^[49]. Spectrum-X is described as scaling in two-tier leaf and spine topologies ^[50]. Converting units, 400 Gb/s is 50 GB/s and 800 Gb/s is 100 GB/s per port, so a job that crosses the fabric instead of staying inside an NVLink domain moves from a path rated in the hundreds of GB/s per GPU to one rated at 50 to 100 GB/s per port. Rated figures are not measured throughput, NVLink ratings are aggregate per GPU while network ratings are per-port line rates, so the two are not like-for-like and the order-of-magnitude contrast is indicative only, and real gaps depend on the collective and message size.

Topograph's own operating numbers are small and documented: a 15 second aggregation delay, a default Kubernetes client limit of 5 queries per second with a burst of 10 ^[6], and a result cache of 100 request IDs ^[12]. NVIDIA's InfiniBand documentation also states that manual topology maintenance is feasible at roughly 32 nodes with a stable network and a careful operator ^[51], and that manual maintenance becomes the dominant source of scheduling misplacement at larger scale. Both are vendor statements rather than measurements.

What independent studies show, and what they do not

No source reviewed measures a Topograph-generated label set against a baseline. The closest evidence concerns topology-aware placement in general, and the setups differ.

• Production LLM training (peer-reviewed). The Arnold scheduler, presented at NeurIPS 2025 ^[52], reports a 10.6% end-to-end gain on more than 9,600 GPUs ^[13], an average speedup of 5.7% and 10.6% at the two scales studied ^[52], and a reduction of up to 1.67 times in communication-group spread in simulation ^[13].
• Sensitivity to spread. The same study measured collective bandwidth falling by up to 17% for collectives and 70% for point-to-point operations as groups spanned more units ^[52], yet found negligible variation of 0.3% within a single unit ^[52]. The gain comes from avoiding wide spread, not from fine leaf-level tuning.
• Communication-aware Slurm allocation. A 2020 workshop paper reports a 9% average reduction in execution time for communication-intensive jobs ^[53] and a 31% reduction in average wait time on three supercomputer job logs ^[53]; these are log-driven results on older systems.
• Simulation. A network-sensitive GPU scheduler reports up to a 69% makespan improvement in a trace-driven simulator ^[54], which is not a hardware result.
• Interference. An SC18 study measured up to a 114% run time increase in the worst case for fixed placements under inter-job interference ^[55].
• Counter-evidence. A 2017 poster on a 3-level fat-tree with a 2:1 taper found minimal placement impact for the applications tested ^[14] and noted that added locality constraints can reduce system utilization ^[14].

The pattern is consistent with a conditional hypothesis: placement matters when jobs are large, communication-bound and the fabric is oversubscribed or shared, and matters little when the fabric is non-blocking relative to the job or the jobs are small. The authors of the studies above did not test Topograph, and their hardware differs from yours. NVIDIA's blog states a qualitative problem (poor placement fragments topology domains), but the text reviewed contains no measured throughput figure for Topograph. For a buyer, the defensible position is that the expected size of any gain is unknown until paired runs on the buyer's fabric are complete.

Meta's engineering description of its RoCE training network explains a related design goal, reducing cross-zone traffic and collective completion time through placement ^[56], which supports the mechanism but not a number for other clusters.

Implications and Future Directions

The adoption gate is operational, not technical. Installing the service is a short exercise; the harder questions are whether the provider's reported topology is accurate, whether refresh covers your change events, and whether a scheduler layer exists that can act on the output. A cluster with a single flat fabric, small jobs and a non-blocking network has little to gain based on the counter-evidence above.

Kubernetes is moving the scheduler layer. Gang scheduling reached beta in v1.37 ^[57] but is off by default, and native topology awareness remains alpha. If a cluster's Kubernetes version and feature gates will not reach that state during the planning horizon, KAI or Kueue is the practical gang layer today. Decisions should be tied to the features actually enabled in your own cluster.

Slurm keeps evolving. Slurm 26.05 introduced ring topologies ^[38], so the set of topology types a generator must emit is not fixed. Slurm-on-Kubernetes deserves its own test. NVIDIA documents Slinky integration that discovers block topology using DRA and GPU Operator labels ^[58]. Because the DRA provider sees only NVLink partitions, a Slinky cluster that spans partitions needs a provider that sees the fabric too.

Independent validation will matter more. NetQ includes topology validation against a user-provided blueprint ^[59], and UFM can compare the current topology with a master topology ^[60]. Pairing such drift checks with Topograph output is a stronger control than either alone, and it also gives auditors evidence of intended versus reported wiring.

The desired-versus-discovered worksheet, the paired-run protocol and the owner matrix give structure, but the evidence that decides adoption has to come from the buyer's own cluster.

Frequently Asked Questions (FAQs)

What is NVIDIA Topograph, and does it schedule jobs?

It discovers cluster topology and translates it into a scheduler's format. It sits between topology sources and schedulers and replaces neither. Placement is performed by Kubernetes, KAI, Kueue or Slurm.

How does Topograph work with Kubernetes topology-aware scheduling?

The Kubernetes engine writes fabric.topograph.run/tier-N and accelerator-domain labels. The default scheduler can use them in affinity rules as a preference, while KAI and Kueue can apply required levels to a gang.

How does Topograph relate to Slurm topology.conf?

The Slurm engine generates tree or block topology. A node up or down trigger can call it, which by design reacts to node events rather than to cabling changes. Slurm 25.05 added per-partition topologies.

Does it work on bare-metal InfiniBand without MNNVL?

Yes through the InfiniBand provider, which uses ibnetdiscover and is the only network topology source when the clique label is absent. It is single-region only.

Which node labels does it write, and what are NFD resources?

Fabric tier labels and accelerator domain labels. The NFD engine instead publishes NodeFeatureGroup resources and is only for consumers of NFD.

How can operators confirm the labels match the wiring after a change?

Diff labels against fresh ibnetdiscover output and the cabling plan after a controlled change, as in the checklist above. Regeneration is triggered by events such as node adds and deletes.

What does the default scheduler lack compared with KAI or Kueue?

It places one pod at a time, so a required gang-wide domain needs a gang-aware layer, where KAI ties PodGroup gang semantics to topology constraints.

Will it improve throughput or lower cost?

That cannot be promised from public sources. Published placement studies show effects from negligible to double digits, under different conditions. Measure with paired runs.

Conclusion

Topograph is best understood as a topology reporting layer with scheduler-specific outputs. Its value is bounded by three factors: the accuracy of the provider's reported topology, the coverage of its refresh triggers, and the existence of a scheduler path that can act on the result. NVIDIA says as much about reported versus intended topology.

For a private cluster, the pilot decision reduces to a short test. Pin a release and record the provider list from that tag. Choose the provider by fabric: NetQ where deployed, InfiniBand otherwise, and do not substitute the DRA provider when the fabric between partitions matters. Choose the scheduler layer by whether jobs need a hard gang-wide domain or only a preference. Run the change-detection checklist and the desired-versus-discovered diff before the paired performance runs. If the diff is clean, the labels track a deliberate wiring change, and paired runs show a difference larger than their variance, adoption is justified by the buyer's own evidence. If any of those fail, the cluster has learned something useful about its topology data before it learned it from a stalled job.