Checkpoint I/O Sizing: Write Bandwidth for 70B-405B

The short answer

Checkpoint write bandwidth depends on three numbers: bytes per checkpoint, how long training may block, and how often you save. With mixed-precision Adam a full training checkpoint is about 12 to 16 bytes per parameter: roughly 0.84 to 1.12 TB for 70B and 4.9 to 6.5 TB for 405B. Required bandwidth is size divided by allowed stall, so a synchronous 5.67 TB save inside 60 seconds needs about 95 GB/s sustained aggregate write. Asynchronous checkpointing blocks the GPUs only while state is copied to host memory, giving storage the whole interval to absorb the write. That turns a peak-bandwidth problem into a sustained-bandwidth and metadata problem.

Totals below are derived from per-parameter byte counts, not measured; check your own checkpoint with du.

What a training checkpoint contains

Per ZeRO, mixed-precision Adam training of Ψ parameters holds an fp16 copy of the parameters and gradients (2Ψ + 2Ψ bytes), plus optimizer states of "an fp32 copy of the parameters, momentum and variance, with memory requirements of 4Ψ, 4Ψ, and 4Ψ bytes". With K = 12 that gives "2Ψ + 2Ψ + KΨ = 16Ψ bytes of memory requirement" ^[1]. Its example: a 1.5B-parameter GPT-2 needs at least 24 GB versus 3 GB for fp16 parameters alone. bf16 is also 2 bytes, so the accounting holds.

A checkpoint is not a dump of GPU memory:

The planning range is 12Ψ (master weights plus moments), 14Ψ (plus bf16 weights, the central case here) and 16Ψ (ZeRO's full in-memory figure, gradients included). Activations are not checkpointed. For the in-memory side, see the LLM training memory calculator.

Checkpoint sizes for 7B, 70B and 405B

These figures are derived arithmetic in decimal units (1 TB = 10^12 bytes):

Example: 405 × 10^9 parameters × 14 bytes = 5.67 × 10^12 bytes = 5.67 TB.

A training checkpoint is 6 to 8 times the weights-only file you would ship to inference, so do not size training storage from model download sizes. Retention multiplies capacity: ten retained 405B checkpoints at 14Ψ take about 57 TB before protection overhead. ByteCheckpoint's operators handle this with a cool-down mechanism that "migrates data from SSD to HDD storage" ^[2].

Why each GPU writes a small shard

The total is large, but under sharded training no single GPU writes much. Llama 3 405B was trained "on up to 16K H100 GPUs". The 16,384-GPU configuration used TP 8, PP 16 and DP 128, with FSDP that "shards optimizer states and gradients" ^[3]. Meta reports that "Checkpointing saves each GPU's model state, ranging from 1 MB to 4 GB per GPU" (same source).

An even split of 5.67 TB over 16,384 GPUs would be about 346 MB per GPU, or 2.8 GB per 8-GPU node. Meta's 1 MB to 4 GB range shows real layouts are uneven (embeddings, pipeline-stage imbalance, replicated tensors), and the largest shard sets a synchronous save's completion time. Size from the shard distribution, not the mean.

Sharding also sets file count. PyTorch Distributed Checkpoint (DCP) "produces multiple files per checkpoint, with at least one per rank", plus a global .metadata file ^[4]. At 16,384 ranks, each checkpoint is a burst of at least 16,384 file creates: a metadata event as much as a bandwidth event.

The stall budget: from checkpoint interval to GB/s

Step 1: pick the interval from the failure rate

In a 54-day snapshot of Llama 3 405B pre-training, Meta counted 466 job interruptions, of which "419 were unexpected" ^[3]. That is one every 54 × 24 / 419 = 3.09 h, about 11,100 s.

Assume (first-order) a failure lands mid-interval on average, losing T/2 of work plus restart time:

At this rate a 30-minute interval wastes about 8% of the cluster, which is why Meta aims "to minimize GPU pause time during checkpointing and increase checkpoint frequency" ^[3]. Your rate will differ; the cluster reliability budget covers estimating it.

Step 2: turn interval and overhead target into a stall budget

A synchronous save blocking S seconds every T seconds costs S/T. For the 405B central case (5.67 TB):

At fixed stall budgets:

These are sustained end-to-end rates at the client, including file creation, fsync and close, not a datasheet sequential-write number. For scale, Meta's Tectonic-based fabric for Llama 3 offered "240 PB of storage out of 7,500 servers equipped with SSDs" with "a sustainable throughput of 2 TB/s and a peak throughput of 7 TB/s", and Meta still calls "the highly bursty checkpoint writes that saturate the storage fabric for short durations" a major challenge ^[3]. A synchronous design buys burst bandwidth that sits idle for 99% of the interval.

Asynchronous save changes which number you size

DCP's async_save "first de-stages the state_dict on to the staging storage (defaults to CPU memory), and then calls the save in a separate thread", and it returns a future ^[4]. In PyTorch 2.15.0 (released 2026-10-28) the docs still mark it "experimental and subject to change", and it runs the save in a thread by default (AsyncCheckpointerType.THREAD; a separate process is optional). The GPU-visible pause is the staging copy plus coordination; the storage write overlaps later training steps.

Published results:

• PyTorch/IBM (PyTorch blog, November 2024): 7B checkpoint downtime fell "from an average of 148.8 seconds to 6.3 seconds", and the GPU-to-CPU phase "can take from 6 – 14 seconds for 7B-13B model sizes" ^[5]. Async "does not compress the actual serialization checkpointing time"; it moves it off the critical path. The team used a separate process group for checkpoint collectives to avoid hangs.
• ByteCheckpoint (research paper, ByteDance production): a "Text Transformer 405B" on 8,960 H800 GPUs (TP 8, DP 70, PP 16, Megatron-LM) showed 0.59 s blocking, 51.06 s end-to-end save and 129.49 s load ^[2]. The paper does not give the checkpoint's byte size, so do not back out a bandwidth.

With async save, size against these constraints instead:

1. Persist bandwidth ≥ size / interval, with headroom. The save must finish before the next starts, or the job blocks or needs a second staging buffer. For 405B at 14Ψ and a 10-minute interval: 5.67 TB / 600 s ≈ 9.45 GB/s minimum; plan 2 to 3 times that (about 20 to 30 GB/s) for contention.
2. Host memory ≥ the node's shard per staging buffer. About 2.8 GB per node for the even 405B split; about 7.7 GB per node for a 70B job on 1,024 GPUs (980 GB / 128 nodes). Small next to server DRAM, but a pinned-memory budget if pinned staging is on: "Pinned memory improves CPU-GPU transfer speeds but uses more memory" ^[4].
3. The write path stays busy most of the interval, competing with dataloader reads. Test under concurrent read load.

Synchronous save at a 1% budget needs hundreds of GB/s for 405B; async needs tens of GB/s sustained plus metadata capacity for the file burst.

Sharded save, deduplication and resharding

Parallel per-rank writes. DCP "supports loading and saving models from multiple ranks in parallel" ^[4]. FileSystemWriter defaults to thread_count=1; raise it first when a node cannot fill its NIC.

Deduplicating replicas. Identical tensors on data-parallel replicas should be written once. Megatron Core's FullyParallelSaveStrategyWrapper exchanges only metadata and, "based on data replication on different ranks, we try to distribute the save as uniformly as possible" ^[6]. Spreading writers matters as much as removing duplicates; if one replica writes everything, a few NICs set the save time.

Load-time resharding. DCP "handles load-time resharding which enables saving in one cluster topology and loading into another" ^[4]. Restarts often run on fewer nodes, and a resharded load reads byte ranges from many files. ByteCheckpoint reports an average 54.20× reduction in checkpoint stalls versus DCP and Megatron MCP baselines, and up to 9.96× faster saves and 8.80× faster loads ^[2]. These are the authors' comparisons on their HDFS backend, not independent results.

Reducing the bytes

• Drop redundant state. Skipping the bf16 copy moves 14Ψ to 12Ψ, about 14% fewer bytes, if your framework can resume from it.
• Incremental and quantized checkpoints. Check-N-Run reduced "the required write bandwidth by 6-17× and the required capacity by 2.5-8×" on Facebook models ^[7]. Those are recommendation models, where only part of the embedding tables changes per interval; dense LLMs update every parameter each step, so do not assume similar gains for 70B to 405B transformers.

Parallel file system design principles

No WEKA, VAST, GPFS or DDN documentation was verified for this article, so this section stays at design-principle level, with the Lustre manual as the concrete reference. Validate any vendor throughput claim with the test plan below.

Aggregate write is a client-side sum. Multiply concurrent writer nodes by what each pushes through its storage NIC, then confirm the servers absorb that sum durably, not just into cache.

Match striping to file layout. Checkpoints are usually file-per-rank, hundreds of MB to a few GB each. Lustre's rule of thumb is to "stripe over as few objects as will meet those needs and no more"; wide striping is for when "the file size is very large and/or is accessed by many nodes at a time", and "small files do not benefit from striping" ^[8]. Thousands of file-per-rank writers with low stripe counts spread across OSTs naturally. For single shared files, Lustre warns that several clients on one stripe "can lead to issues with lock exchange", which overstriping (Lustre 2.13+) addresses.

Metadata is a separate budget. One file per rank per checkpoint means tens of thousands of creates, fsync calls and closes in seconds, and retention deletes add the same again. ByteCheckpoint's team had to address a "QPS bottleneck of HDFS NameNode metadata requests" ^[2]. Test create rates at full rank count, not just GB/s.

Separate the hot tier from retention. Keep recent checkpoints on flash near the cluster and age older ones to cheaper capacity. A node-local NVMe tier that drains to shared storage protects the stall budget, but a lost node takes its unsynced shard with it, so decide explicitly which copy is your recovery point. For the cloud cost side, see AWS P5 storage cost.

The restart path is a read problem

After a failure every rank loads at once, and the cluster idles until the slowest rank has its state: 5.67 TB in the central 405B case, more if replicas read the same weights or resharded loads read overlapping ranges. ByteCheckpoint's 405B job took 129.49 s to load against 51.06 s to save ^[2]. At Llama 3's interruption rate that load happens more than seven times a day, so qualify restart read bandwidth too. If restores go straight into GPU memory, the GDSIO qualification runbook covers the read path.

Worked example: 405B on 16,384 GPUs

Illustrative assumptions: dense 405B, bf16 with fp32 master and Adam, 14Ψ, 2,048 nodes × 8 GPUs, Llama 3 interruption rate, checkpoint overhead ≤ 1%.

1. Checkpoint size: 405 × 10^9 × 14 B = 5.67 TB. Even shard: 5.67 TB / 16,384 ≈ 346 MB per GPU, about 2.8 GB per node.
2. Interval: a 10-minute interval holds expected lost work to about 2.7% of wall clock (300 s / 11,135 s).
3. Synchronous option: 1% of 600 s = 6 s, so 5.67 TB / 6 s = 945 GB/s aggregate, about 462 MB/s per node (945 / 2,048). Per node that is easy; the aggregate burst plus about 16,000 file creates in the same seconds is not. Relaxing to a 60 s stall (10% overhead at this interval) still needs 94.5 GB/s.
4. Asynchronous option: blocking is the staging copy, published at sub-second (ByteCheckpoint, 405B) to 6 to 14 s (PyTorch blog, 7B to 13B); measure yours. Storage must persist 5.67 TB within 600 s: ≥ 9.45 GB/s, about 20 to 30 GB/s with headroom. Host staging: about 2.8 GB per node per buffer.
5. Restart read: loading 5.67 TB in 2 minutes takes about 47 GB/s aggregate read.
6. Capacity: 10 retained checkpoints ≈ 57 TB on the hot tier, before protection overhead.

Async needs roughly 30 to 50 times less write bandwidth than 1%-overhead synchronous save; remaining risk is metadata and reads.

Test plan and checklist

Model the real pattern: thousands of writers, one file each, a burst, silence, then a mass read. IOR fits: -F makes each MPI process "perform I/O to a unique file", -b sets per-task block size, -t transfer size, -e will "perform fsync upon POSIX write close", and -C reorders tasks so readback avoids the client's own cache ^[9].

1. Set -b to your measured per-rank shard size, including the largest shard.
2. Run one rank per GPU across as many nodes as you can spare with -F -e -w; record aggregate GB/s and the slowest rank's time.
3. Repeat while another job saturates reads, approximating async save under training.
4. Read back with -r -C for restart bandwidth.
5. Measure file create/close rates at full rank count separately.
6. On the real framework, time DCP or Megatron saves with async on and off, and check du against the 12Ψ/14Ψ/16Ψ assumption.
7. Test a resharded restore onto fewer nodes, the restart you will actually perform after hardware loss.

Checklist: checkpoint bytes measured · interval derived from your failure rate · stall budget written down · sync vs async decided · persist bandwidth ≥ size / interval with headroom · metadata rate tested · restart read tested with resharding · retention tier sized.

GPU Smith helps teams work through this kind of storage sizing and its test plans when specifying training clusters.