Storage requirements for model training and checkpointing come from three kinds of data, and each uses storage in its own way. Datasets are written once per version and read again on every pass. Checkpoints, the saved training state a run resumes from, are written in bursts while training waits and read back after a failure. Model weights are kept as files that evaluation and later training read.
Checkpoints are where size and timing meet. In the rules of the MLPerf Storage benchmark, the checkpoint of a 70-billion-parameter model is 912 GiB, about 14 bytes per parameter and roughly seven times its 16-bit weights (computed here from [1, 9]). For pre-training a 405-billion-parameter model on up to 16K H100 GPUs, its developers describe a 240 PB storage fabric that sustains 2 TB/s and peaks at 7 TB/s, and name bursty checkpoint writes as a major challenge [5].
Storage requirements for training and checkpointing
| Data | What it holds | How it is written | How it is read | Sources |
|---|---|---|---|---|
| Datasets | Samples or tokens in files and shards | Once per version | On every epoch, in random batches | [7, 20, 22, 24] |
| Checkpoints | Weights, optimizer state and training progress | In bursts, as files from every process in parallel | After a failure, and by evaluation and post-training jobs | [3, 4, 5, 7, 16] |
| Model weights | Parameters, at 2 bytes each in 16 bits or 1 in FP8 | As files, in safetensors with a header listing each tensor | By evaluation and later training | [1, 2, 16, 28] |
NVIDIA's DGX SuperPOD reference architecture expects most deep learning workloads to be read-dominant, with peak reads and writes for LLM checkpoints, and advises characterizing workloads before fixing performance and capacity [24].
Training data storage: datasets read again and again
"The key I/O operation in DL training is re-read," NVIDIA's reference architecture states, because training reuses the same data on every pass [24]. Dataset sizes vary with the source and the format. The FineWeb dataset card lists more than 18.5 trillion tokens of cleaned English web text in 54.8 TB of files, as viewed on 7 November 2026 [20]. Common Crawl's October 2026 archive holds 371.19 TiB of uncompressed content, and its extracted text files take 5.85 TiB compressed, about 1.6% of that (computed here from [21]).
When a dataset outgrows memory, reads can hold up the GPUs. In a 2021 study on V100 servers, models spent 10% to 70% of epoch time waiting on blocking I/O, despite prefetching, because the GPUs computed faster than storage could fetch [18].
File size changes the load. In MLPerf Storage v3.0, one training workload reads files of about 140 MiB and needs about 5.9 GiB/s for each simulated B200 accelerator. The other reads JPEG files of about 315 KiB at about 155 MiB/s, roughly 500 files a second, which makes it a load of metadata and small operations [7]. Packing many small samples into large tar shards turns random reads into sequential ones, which the WebDataset project puts at 3 to 10 times faster on local drives [22].
GPU servers also carry local flash. A DGX B200 lists eight 3.84 TB NVMe drives in RAID 0 as a data cache, 30.72 TB per system (computed here from [26]), and NVIDIA describes local NVMe as a place to cache or stage data [24]. In a 2025 study of one operator's research clusters, inputs, outputs and checkpoints are expected to be durable and decoupled from the lifetime of the job [10].
LLM checkpoint size and write bursts
A checkpoint that can resume training holds more than the weights. PyTorch's documentation says to save the optimizer's state and progress such as the epoch, and that such a checkpoint is often 2 to 3 times larger than the model alone [3]. CheckFreq also saves the data loader's position through a resumable iterator [11].
For mixed-precision training with the Adam optimizer, the ZeRO paper counts 2 bytes per parameter for 16-bit weights, 2 for gradients and 12 for the optimizer's 32-bit copy of the weights, momentum and variance [1]. MLPerf Storage's checkpoint sizes match weights plus optimizer state: 105 GiB for 8 billion parameters, 912 GiB for 70 billion and 5.29 TiB for 405 billion, each about 14 bytes per parameter (computed here from [1, 9]). PyTorch's ratio of 2 to 3 times comes from a different setting [3].
Checkpoints are many files written at once. PyTorch's distributed checkpoint writes at least one file per rank, or process, and can load into a different cluster layout [4]. MLPerf's 405-billion-parameter checkpoint is 1,544 files from 512 processes, about 10 GiB per process (computed here from [7, 9]), with an fsync forcing every file to durable media before it counts as written [7].
While a checkpoint is written synchronously, training stalls, and the write takes about its size divided by bandwidth [7]. At the 20 GBps and 62 GBps write guidelines NVIDIA gives for one scalable unit of 32 DGX B200 systems, a 912 GiB checkpoint takes about 49 and 16 seconds (computed here from [9, 24, 25]).
How often to checkpoint during LLM training
Failures set the interval. In a 54-day period of the 405-billion-parameter pre-training, its developers counted 419 unexpected interruptions [5]. The 2025 study of research clusters projects a mean time to failure of 1.8 hours for jobs on 16,384 GPUs [10]. It restates the Daly and Young rule: the optimal interval is about the square root of twice the checkpoint write overhead divided by the job's failure rate [10]. For about 12,000 GPUs at the failure rate measured on its cluster RSC-1, a 5-minute write gives an interval of about 38 minutes and a 10-second write about 7 minutes (computed here from [10]). Shorter writes allow more checkpoints and less lost work.
Asynchronous checkpointing copies state from GPU to host memory, lets training continue and writes to storage in the background. In PyTorch's test, a 7-billion-parameter model's pause fell from 148.8 seconds to 6.3 seconds [14]. The method does not shorten the serialization itself. It moves the final write off the training path, to CPU threads [14]. GEMINI, a 2023 in-memory checkpointing system, keeps its failure-recovery checkpoints in host CPU memory and reaches a checkpoint every iteration [15].
Restores are a read burst. MegaScale has one worker per data-parallel group read its partition from storage and broadcast it to the others, which reduces the load on storage [13]. The research on training on spot GPUs shows what lost work and checkpoint overhead cost when capacity can be taken back.
Model weights by version
Weights take 2 bytes per parameter at 16 bits and 1 byte in FP8, so a 70-billion-parameter model is about 140 GB or 70 GB (computed here from [1, 2]). The article on GPU memory for LLM inference gives 4-bit sizes and the sizes of published checkpoint files.
Checkpoints and weights have readers other than a restart. ByteCheckpoint's authors report that evaluation tasks download new checkpoint files shortly after they are written, that access then drops, and that every file is kept for traceability, so files move from SSD to hard drive servers [16]. In a 2024 datacenter study, an evaluation job on HumanEval spent 29.5% of its time loading the model and preprocessing data before GPU inference began [17].
The safetensors format puts a header before the tensors that lists each one's type, shape and offsets, so a reader can inspect a file without loading all of it, and a downloaded file can be loaded without running arbitrary code [28]. In a versioned bucket, every object added receives a unique version ID [31], and FineWeb keeps its previous versions available in named branches [20].
Which storage type fits which data
Published clusters mix storage types.
| Cluster or system | Storage reported | Sources |
|---|---|---|
| One operator's research clusters | A flash-backed POSIX file system over NFS, a read-only dataset cache and an object store | [10] |
| The same operator's two 24,576-GPU clusters | A flash distributed store for checkpoints and data loading, and a parallel NFS for code | [6] |
| A 2024 datacenter study | An all-NVMe shared parallel file system, and an extra network adapter for storage on one cluster's nodes | [17] |
| MegaScale and ByteCheckpoint | HDFS, with SSD and hard drive tiers in ByteCheckpoint | [13, 16] |
Object storage reaches training code through the S3 API. Amazon's S3 connector for PyTorch reads datasets and saves distributed checkpoints to Amazon S3 without writing to local storage first [23]. MLPerf Storage v3.0 added an S3 access layer, and about one-sixth of its submissions used it [8].
Ceph serves object, file and block storage from one cluster: an S3-compatible object gateway with bucket versioning, multipart uploads and lifecycle rules [29, 30], a POSIX file system whose clients read and write file data directly [32], and thin-provisioned block devices with snapshots [33].
What a storage sizing needs
A sizing starts from the work: the dataset's bytes and file sizes, the GPUs that read it and their read rate, the checkpoint's bytes per parameter, the interval and how many checkpoints are kept, the failure rate at that scale, and the network between storage and GPUs. The explainer on DGX, HGX and MGX shows what storage and networking come with each kind of GPU system.
Rillor's distributed storage work designs and builds Ceph clusters for datasets, checkpoints and model weights, on the client's servers, on storage nodes Rillor supplies or inside a private superintelligence (SI) cloud.
Read the full report
The full report adds the checkpoint arithmetic for each MLPerf size, failure rates and interval examples, read and write rates per GPU, restores and other checkpoint readers, and all 35 sources.