Spot instances for LLM training cost less per hour, and the research shows the saving left at the end of a run is smaller than the posted discount. In Can't Be Late (NSDI 2024), a RoBERTa training job on one V100 spot instance priced 67% below on-demand cost 41% to 48% less overall, once its policy switched to on-demand to meet a deadline [25]. Bamboo (NSDI 2023) reported costs 2.4 times lower than on-demand across six models on V100 spot clusters [22].
The difference comes from work lost since the last checkpoint, time spent restarting, and on-demand capacity bought when spot runs out [20, 22, 25]. Most training studies ran on V100-era instances from 2021 to 2024, and none reviewed here measures net savings for multi-node pre-training on H100 or newer spot capacity, so hardware and year go with every number.
How spot GPU capacity works at each provider
Each provider sets its own terms (viewed 24 October 2026).
| Provider | Notice before reclaim | Discount, as the provider states it | Interruption data published |
|---|---|---|---|
| AWS Spot | Two minutes, best effort [2] | Up to 90% [6] | Frequency bands per instance pool, from under 5% to over 20% [6, 7] |
| Google Cloud Spot VMs | 0 seconds by default or 120 seconds, then up to 30 seconds of shutdown, best effort [10] | Up to 91% [10] | Daily share of stopped Spot VMs that were preempted, excluding N1 GPU VMs [11] |
| Azure Spot | Up to 30 seconds, best effort [14, 15] | Up to 90% [16] | Hourly eviction rate from the last 7 days [14] |
| Oracle preemptible | Two minutes [17] | 50% less, fixed [17] | Not in its documentation [17] |
| Vast.ai interruptible | Not stated. Instances are paused and data is kept [19] | "often 50%+ cheaper than on-demand" [19] | Not in its documentation [19] |
AWS and Azure call their notices best effort, as Google does its shutdown period, and AWS says an instance can be interrupted before the warning arrives [2, 4, 10, 14]. Lost compute is billed: AWS charges for the seconds used when it interrupts an instance after its first hour [5], and Google waives charges only for Spot VMs preempted within their first minute [10]. For spot prices next to other terms, see what posted GPU cloud prices tell you.
Spot GPU interruption rates and what they measure
The three metrics do not convert into one another [6, 11, 14]. AWS gives a frequency band over the last 30 days [6]. Azure gives the chance of eviction within the next hour [14]. Google divides the Spot VMs preempted in a day by all Spot VMs that stopped that day, including stops by users [11].
AWS's advisor data file, last modified 20 March 2026, lists these Linux bands for 8-GPU instances in us-east-1 [7, 8]:
| Instance | GPUs | Interruption band | Savings over on-demand |
|---|---|---|---|
| p5.48xlarge | 8 x H100 | Under 5% | 57% |
| p5en.48xlarge | 8 x H200 | Under 5% | 67% |
| p6-b200.48xlarge | 8 x B200 | 10% to 15% | 79% |
| p4d.24xlarge | 8 x A100 | 15% to 20% | 56% |
Preemptible GPU training and checkpointing
A checkpoint sets how much work an interruption destroys. In CheckFreq (FAST 2021), ResNext101 on one V100 takes 270 hours, about 3.9 hours per epoch, so with checkpoints at epoch boundaries an interruption wastes about two hours of GPU computation on average [20]. Checkpointing at the level of iterations cut recovery from hours to seconds, with runtime overhead within 3.5% [20].
When Bamboo's authors trained GPT-2 on 64 EC2 spot instances with checkpoint and restart, restarting and redoing work took 77% of the training time [22].
Restarts cost time too. Can't Be Late measured about 0.3 hours per changeover on a p3.2xlarge, for provisioning, setup and lost progress [25]. SkyNomad measured a cold start of about 6 minutes for fine-tuning jobs with 100 GB and 500 GB checkpoints [30].
Asynchronous checkpointing helps. PyTorch reported a 7B model's checkpoint downtime falling from 148.8 seconds to 6.3 seconds, with the write to storage finishing in the background [34]. A checkpoint is safe only once that write ends, which matters under a 30-second notice [14, 34].
A classic rule, restated by Meta researchers at HPCA 2025, puts the optimal interval at about the square root of twice the checkpoint write time multiplied by the mean time to failure [28].
Redundancy and fast recovery in distributed training
Four systems papers reduce or replace checkpoint restarts.
| System and conference | Setup | Result as reported |
|---|---|---|
| Varuna, EuroSys 2022 | Azure low-priority V100 VMs, GPT-2 8.3B | 5.85 times the cost-performance of a 16-node DGX-2 cluster [21] |
| Bamboo, NSDI 2023 | EC2 single-V100 spot instances, traces replayed at 10%, 16% and 33% hourly preemption | 3.7 times the throughput of checkpointing, costs 2.4 times lower than on-demand [22] |
| Oobleck, SOSP 2023 | 30 A40 GPUs, 200 Gbps InfiniBand | Up to 29.6 times Varuna's throughput on GPT-3 6.7B with a failure every 10 minutes, up to 9.1 times on average on spot traces [23] |
| Parcae, NSDI 2024 | AWS p3.2xlarge trace replayed on 32 V100s | 2.59 times Varuna's and 3.0 times Bamboo's throughput on average, 3.24 times cheaper than on-demand [24] |
Headline numbers depend on metric and setup. Bamboo's paper gives 2.4 times lower cost in its abstract, 3.6 times in its introduction and 1.95 times higher value (throughput per dollar) in its evaluation [22]. Bamboo ran out of memory on every GPT-3 model in Oobleck's tests, so the 29.6 times compares against Varuna only [23]. Oobleck also found Bamboo slower than Varuna, reversing Bamboo's result. Its authors attributed the gap to their 200 Gbps network and NVMe storage, against up to 10 Gbps and S3 in Bamboo's tests, and to Bamboo running Varuna with Bamboo's batch settings, which throttled Varuna [23].
Spot vs on-demand GPU cost under a deadline
A job with a deadline buys on-demand capacity when spot disappears. Can't Be Late collected three months of availability traces in nine AWS zones for V100 and K80 instances and found availability could fall from 100% to 0% within hours [25]. Its savings followed the posted discount [25].
| Workload | Spot discount | Saving, tight deadline | Saving, loose deadline |
|---|---|---|---|
| ML training, RoBERTa on 1 x V100 (AWS) | 67% | 41% | 48% |
| Bioinformatics, CPU (Google Cloud) | 91% | 63% | 84% |
| Data analytics, CPU (AWS) | 55% | 27% | 32% |
Source: [25], Tables 4 and 5.
SkyNomad, a January 2026 preprint, adds H100 evidence. For a job on 16 single-H100 spot instances on Google Cloud that stops when any one is preempted, median lifetimes ranged from under 1 hour to over 20 hours by zone [30]. In fine-tuning runs on AWS L4, A10G and A100 instances, its multi-region policy saved 10% to 55% against the "best single-region baseline", with egress for moving checkpoints at up to 16% of total cost [30]. One region had no spot capacity for more than 70% of the time, and on-demand fallback made up 74% of its cost [30].
SageMaker managed spot training reports savings that leave out intervals lost in their first hour and time spent waiting for capacity, and its jobs stop at a set limit without switching to on-demand [9].
Fine-tuning on spot instances
The Vicuna team reported in 2023 that managed spot instances cut the cost of training its 7B model on 8 A100 GPUs from $500 to around $140 [35]. The SkyNomad runs above are fine-tuning jobs too [30].
Interruptions without spot: a large-run baseline
Dedicated capacity stops too. Meta reported 466 job interruptions in a 54-day snapshot of Llama 3 405B pre-training on up to 16K H100 GPUs. Of these, 47 were planned and 419 unexpected, with GPU issues behind 58.7% of the unexpected ones [32]. Effective training time still exceeded 90% [32]. In Meta's A100 research clusters, mean time to failure was 7.9 hours for 1,024-GPU jobs and 47.7 days for 8-GPU jobs [28].
Serving and rollout on spot capacity
SpotServe (ASPLOS 2024) cut serving cost by up to 54% against on-demand on T4 instances, with average latency up to 18% higher and P99 latency up to 90% higher [26]. RLBoost, a preprint first posted in October 2025, ran reinforcement learning rollout on spot H100 instances, kept training on non-preemptible ones and improved cost efficiency by 28% to 49% [31].
Is spot worth it for LLM training?
The studies found net savings under these conditions [20, 22, 23, 25, 30, 31]:
- a posted discount large against the cost of lost and repeated work
- checkpoints that write and restore in minutes or less, on fast storage
- slack between expected runtime and the deadline
- capacity drawn from several zones or regions
- stateless work kept apart from stateful training
To set a year of rent against the hardware's price, see rent-to-value.
Rillor's GPU and neocloud pricing method keeps spot and preemptible prices apart from on-demand and reserved prices, and each observation in its schema carries the GPU class, region, term, source and UTC capture time.
Read the full report
The full report adds each paper's hardware and traces, provider billing terms, checkpoint arithmetic and where sources disagree.