How much GPU memory LLM inference needs comes down to two terms and a reserve. The weights take a fixed amount: parameters times bytes per parameter. The KV cache grows with every token that each active request holds. The serving engine also reserves memory for activations and buffers [18, 21].
One published 70B model's checkpoint in BF16 is 131.42 GiB [27]. Its KV cache takes 320 KiB per token in BF16, so one 128K-token sequence (131,072 tokens) holds 40 GiB [1, 14]. At vLLM's default memory setting, BF16 weights fit on no single 80, 94 or 141 GB GPU. FP8 weights fit on each, with little room left on the 80 GB card, and the KV cache then decides how many users and how much context each GPU serves [17, 27, 28, 33, 34, 35, 36].
Worked figures below are arithmetic from the cited sources. None is measured.
How much GPU memory LLM inference uses
GPU memory = weights + KV bytes per token x tokens held by all active requests + engine reserve
Weights take about 2 bytes per parameter in BF16, 1 in FP8 and about half a byte at 4 bits, which puts a 70B model near 140, 70 and 35 GB [16, 31, 32]. In the PagedAttention paper's profile of a 13B model on a 40 GB A100, about 65% of memory held weights and close to 30% held the state of requests [9].
VRAM for a 70B model: the weights
Published checkpoints confirm the weight rule.
| 70B checkpoint | Size of the files | Share of BF16 |
|---|---|---|
| BF16, 70,553,706,496 parameters | 131.42 GiB | 100% |
| FP8 weights and activations | 67.68 GiB | 51.5% |
| INT4 weights, group size 128 | 36.81 GiB | 28.0% |
Sources: [27, 28, 29], viewed 24 October 2026. BF16 is the model maker's own release. FP8 and INT4 are published checkpoints of another 70B release by the same maker, with the same architecture. Shares are this note's arithmetic.
The quantized files sit above one half and one quarter because the embeddings and output layer stay in 16 bits, and the 4-bit file also stores a scale for each group of 128 weights [28, 29]. Their publisher reports MMLU scores of 81.31 for FP8 and 80.62 for INT4, against 81.60 for that release unquantized [28, 29]. The 70B model's authors warn that standard benchmarks can miss FP8 errors [1]. Mixture-of-experts models are sized by total parameters: one with 13B active per token has memory costs proportional to its 47B total [12].
KV cache size calculation
The KV cache stores one key vector and one value vector per layer and per KV head for each token held [5, 14]:
KV bytes per token = 2 x layers x KV heads x head size x bytes per element
The 70B model has 80 layers, a model dimension of 8,192, 64 attention heads and 8 key/value heads [1]. Its maker's reference code sets head size to model dimension over attention heads, 128 here, and sizes the cache by KV heads [14]. In BF16 that is 2 x 80 x 8 x 128 x 2 = 327,680 bytes, or 320 KiB per token [1, 14]. A sequence of 8,192 tokens holds 2.5 GiB and one of 32,768 holds 10 GiB [1, 14].
Two published versions of the formula need care. NVIDIA's 2023 technical blog on inference writes it with heads times head size, equated with the model's hidden size [31]. That holds only for multi-head attention. For the 70B model it gives 2.5 MiB per token, eight times the figure above [1, 31]. Hugging Face's release post for the same model family prints a 405B row twice what the formula gives for 126 layers and 8 KV heads, by this note's arithmetic [1, 32].
Attention design changes the cache several-fold
Multi-query attention shares one key and value head across all query heads [2]. Grouped-query attention keeps more than one KV head and fewer than the query heads, with quality close to multi-head attention [3]. The 70B model uses 8 KV heads for 64 query heads, one eighth of the multi-head cache, to reduce the size of key-value caches during decoding [1, 3].
A 236B mixture-of-experts model with latent attention compresses keys and values into a latent vector, and its paper reports a KV cache 93.3% smaller than that of the same maker's earlier 67B model [5]. The deployed model also stores each cache element in about 6 bits on average [5]. By this note's arithmetic from both papers' tables, the 93.3% matches that 6-bit cache, and the latent design alone at 16 bits gives about 82% less [5, 6]. Sliding windows cap the cache at the window size [7]. In one 7B model, a rolling buffer cuts cache memory 8 times on a 32K sequence [7].
Context length and concurrent users
The cache is unique to each sequence, so it grows with every active request [4]. For the 70B model in BF16, ten users at 8K tokens each hold 25 GiB, by this note's arithmetic [1, 14]. At 32K each they hold 100 GiB, more than the 67.68 GiB of FP8 weights [28].
The PagedAttention authors measured that earlier systems used only 20.4% to 38.2% of their KV cache memory for token states [9]. Paged blocks brought waste near zero and raised throughput 2 to 4 times at the same latency [9]. When the cache still runs out, vLLM preempts requests and recomputes them, at a cost in latency [18].
FP8 KV cache memory and its caveats
An FP8 cache halves the KV term, to 160 KiB per token for the 70B model [1, 14]. Without calibration, vLLM sets every scale to 1.0. It recommends calibrated scales and names sliding-window layers as more sensitive to KV quantization [15].
vLLM's April 2026 evaluation used uncalibrated scales. On the other 70B release, the MRCR long-context test to 128K tokens recovered about 97 to 98% of the baseline AUC [30]. The same work found a Hopper kernel problem that cut 128K needle-in-a-haystack accuracy from 91% to 13% until a fix restored it [30]. The authors advise staying with BF16 for contexts under about 7K tokens, for models with many small sliding-window layers, and where uncalibrated accuracy falls below 95% on the workload [30].
GPU sizing for LLM serving on one GPU
NVIDIA lists 80GB for the H100 SXM, 94GB for the H100 NVL and 141GB for the H200, and marks the H200 figures as preliminary [23, 24]. The driver reports totals close to those labels in binary units: 81,559 MiB (79.65 GiB), 95,830 MiB (93.58 GiB) and 143,771 MiB (140.40 GiB) [33, 34, 35]. The test below works in GiB on both sides, with the default of 0.92 of GPU memory in vLLM's stable documentation as the budget [36]. Its development documentation, viewed 7 November 2026, gives 1.0 when the extensible KV cache is on [17].
| GPU | Budget at 0.92 | BF16 weights | Left after FP8 weights | BF16 KV tokens | FP8 KV tokens |
|---|---|---|---|---|---|
| H100 SXM 80GB | 73.28 GiB | Do not fit | 5.60 GiB | about 18,300 | about 36,700 |
| H100 NVL 94GB | 86.10 GiB | Do not fit | 18.42 GiB | about 60,400 | about 120,700 |
| H200 141GB | 129.17 GiB | 2.25 GiB short | 61.49 GiB | about 201,500 | about 403,000 |
This note's arithmetic from [17, 27, 28, 33, 34, 35]. The space left also holds activations and CUDA graphs, so the token counts are ceilings.
By the same arithmetic, with FP8 weights, one H200 holds at most about 24 sequences of 8K tokens with a BF16 cache, or 49 with an FP8 cache [17, 28, 35]. An H100 80GB holds about two [17, 28, 33]. The reserve depends on the engine. vLLM profiles memory use and estimates CUDA graph memory at startup [18]. TensorRT-LLM gives the KV cache 90% of the memory still free after loading [21].
How many GPUs to serve a 70B model
Tensor parallelism splits the weights and the KV heads across GPUs [18, 20, 21]. vLLM divides KV heads among GPUs until the GPU count reaches the KV head count, then copies them [20]. For the 70B model in BF16 on H100 80GB GPUs:
| Split | Weights per GPU | Left per GPU | KV per token per GPU | KV tokens for the group |
|---|---|---|---|---|
| 2 GPUs | 65.71 GiB | 7.57 GiB | 160 KiB | about 49,600 |
| 4 GPUs | 32.85 GiB | 40.42 GiB | 80 KiB | about 530,000 |
| 8 GPUs | 16.43 GiB | 56.85 GiB | 40 KiB | about 1.49 million |
This note's arithmetic from [1, 17, 20, 27, 33], with weights split evenly.
By the same arithmetic, two H200s leave 63.46 GiB each, about 415,900 tokens for the group [1, 17, 20, 27, 35]. vLLM's guidance is one GPU when the model fits, tensor parallelism inside a node when it does not, and pipeline parallelism added across nodes [19]. Newer parts carry more memory per GPU [25, 26]. The parts inside an eight-GPU HGX server are set out in what an HGX B200 or B300 server price includes. Renting the GPUs instead is covered in what posted GPU cloud prices tell you.
Sizing the hardware, platform and models is the Design step when Rillor deploys a private superintelligence (SI) cloud: a dedicated environment with your models and data on infrastructure you control.
Read the full report
The full report, with the KV table for the family's 8B, 70B and 405B models, the attention designs side by side, the 4-bit and multi-GPU cases on all three GPUs and the complete source list, is available as a PDF.