Send the same prompt to the same model with the same settings twice, and the two answers can differ. LLM nondeterminism is that variation: output that changes for a request that has not changed. Part of it is by design: one provider's API sets the default temperature at 1, and another uses a random seed unless the caller fixes one, so at default settings each token is drawn at random [14, 15]. The rest survives when sampling is switched off, and it comes from how inference servers do arithmetic, how they batch requests and how a model is served over time.
In a September 2025 experiment on an open-weights model, 1,000 runs of one prompt at temperature 0 produced 80 different completions [1]. A 2024 study of five models from different families, each run ten times at temperature 0, top-p 1 and a fixed seed, found accuracy varying by up to 15% between runs [2]. For anyone testing an agent before it goes live, a single run is one sample from a spread of possible answers.
Where LLM nondeterminism comes from
At each step a language model assigns a probability to every token in its vocabulary, and a decoding rule picks one. Sampling draws at random from that distribution, so any token with a non-zero probability can be chosen [28]. Temperature reshapes the distribution: one provider's API reference describes higher values as more random and lower values as more focused [14]. Top-p, or nucleus sampling, keeps only the tokens in the top share of the probability mass [14, 27]. Greedy decoding skips the draw and always takes the most probable token, which is what temperature 0 means [1, 15].
Sampling explains why answers vary at default settings. The causes that survive greedy decoding fall into four groups.
| Cause | What can differ between two identical requests | Sources |
|---|---|---|
| Floating-point order | The order in which the GPU adds numbers | [1, 20] |
| Batch composition | How many other requests share the server's batch | [1, 33] |
| Hardware and precision | GPU type, GPU count, number format, tensor-parallel size | [3, 34] |
| Serving stack | Router, sampling code and compiler behind one model ID | [11, 24] |
Temperature 0 is not a determinism switch
One model provider's glossary states that results are not fully deterministic even with temperature set to 0, and that identical inputs can produce different outputs across API calls [12]. Another calls temperature-0 responses mostly deterministic, with a small amount of variation still possible [15].
Measured figures agree. A study of a widely used hosted chat model on 829 code-generation problems, five requests per problem, counted the problems where the five programs produced no equal test output [5]. On CodeContests that share was 75.76% at the default temperature of 1 and 43.64% at temperature 0 [5]. On HumanEval it fell from 47.56% to 18.29% [5]. A January 2026 preprint found a unique output fraction of 0.240 for a hosted model and 0.093 for an 8-billion-parameter open-weights model on exact repeats at temperature 0, against 0.987 for both at 0.7 [32].
Some model families take the setting away. One provider's documentation states that its newest families ignore custom temperature, top-p and top-k values [16].
Batch invariance: the load on the server is part of the input
The usual explanation is floating-point arithmetic. Rounding A + B and then adding C can give a different result from adding A to the rounded B + C, so computations that look identical can differ even when every operation follows the IEEE 754 standard [20].
The September 2025 post "Defeating Nondeterminism in LLM Inference" argues that this is only half the story [1]. The post finds that the GPU routines (kernels) in a language model's forward pass, matrix multiplication included, give the same bits on every run for the same input [1]. What they lack is batch invariance: a result for one request that stays the same however many other requests are processed alongside it [1]. Servers batch requests for speed, and kernels' reduction orders vary with batch size, so the order of additions for one user's request shifts with the load [1, 33]. From one user's side, the other users are a hidden part of the input. The post's authors argue that varying load is the main reason nearly all inference endpoints are nondeterministic [1].
Their experiment shows the effect. All 1,000 completions of "Tell me about Richard Feynman" matched for the first 102 tokens [1]. At token 103, 992 continued with "Queens, New York" and 8 with "New York City" [1]. With kernels rewritten so that each element's reduction order stays fixed whatever the batch size, all 1,000 completions were identical [1].
Hardware, precision and expert routing
A NeurIPS 2025 study ran open-weights models of 7 to 8 billion parameters under 12 configurations: two GPU types (L40S and A100), two GPU counts and three batch sizes [3]. Under the 16-bit bfloat16 (BF16) format with greedy decoding, a reasoning model distilled from a larger one showed a standard deviation of up to 9% in accuracy on AIME'24, and response lengths that differed by up to 9,000 tokens [3]. Under 32-bit floating point (FP32), results were near-perfectly reproducible [3]. In reasoning models, the authors note, minor rounding differences in early tokens can cascade into divergent chains of thought [3].
Running a model across a different number of GPUs moves results too. A 2025 paper found that identical inputs gave different outputs when the tensor-parallel size changed, even under greedy decoding, and built kernels that give bit-wise identical results regardless of that size [34].
Sparse mixture-of-experts models add a mechanism of their own. Under capacity limits, tokens from different sequences compete for places in expert buffers, so such a model is deterministic only at the level of the batch [25]. Tokens that overflow an expert are not processed by that layer [26]. No source reviewed shows this in a particular hosted model. A code-generation study lists it, for the hosted model it tested, only as one hypothesis [5].
The same model ID, a different serving stack
One model provider's documentation says each ID identifies a pinned version with fixed weights, while the serving infrastructure around it, including the request router, safety classifiers and sampling logic, can change, occasionally with minor differences in behavior [11]. Aliases move further: another provider's latest aliases are hot-swapped with each new release, with two weeks' notice [17].
Infrastructure changes can also be faults. A model provider's September 2025 postmortem describes three infrastructure bugs that intermittently degraded responses between August and early September 2025 [24]. A context window routing error reached 16% of one model's requests in the worst hour, on 31 August, and an approximate top-k operation sometimes returned wrong results, only for certain batch sizes and model configurations [24]. The provider's evaluations did not capture the degradation users reported [24].
Version changes are a separate effect. Two dated versions of one widely used hosted model, three months apart in 2023, scored 84% and 51% on the same prime-number questions [6].
Can you make LLM output deterministic?
Partly, and at a cost. Hosted seeds are best effort. One provider's 2023 guide said outputs are mostly identical when the seed, the parameters and a system fingerprint match, with a small chance that they differ [13]. Its current API reference marks both the seed and the fingerprint deprecated and states that determinism is not guaranteed [14]. Another provider says the same of its seed [15].
Open inference engines go further. vLLM offers a batch-invariant mode, and its documentation limits reproducibility to the same hardware and the same vLLM version [22, 23]. Over 50 trials on an 8-billion-parameter open-weights model, SGLang's deterministic mode returned one unique output in every test, against up to 18 in normal mode, and in offline tests most of its slowdowns fell between 25% and 45% [21]. A January 2026 preprint makes determinism a per-request option, so its cost scales with the share of traffic that asks for it [33].
What it means for testing an agent
Agents compound the effect. A January 2026 guide from one model provider notes that a task that passed on one run might fail on the next [31]. In an April 2026 preprint at temperature 1.0, agents picked the same tools in the same order with a mean similarity of 0.87, while the consistency of their arguments averaged 0.69 [35]. pass^k, the chance that all k trials succeed, scores whether an agent succeeds every time [10]. Agent evaluation metrics shows how it is computed.
The sources point to three habits. Score repeated runs with error bars, and do not lower the temperature only to reduce variance, which a 2024 paper on error bars advises against unless the new temperature is the one under study [8]. Record the prompts, code, generation settings and model version behind every score [9, 30]. Test rephrasings as separate inputs, since formatting changes alone moved one open-weights model by up to 76 accuracy points [7]. Evaluations often score a single output per example [4]. What agent benchmarks measure covers how public agent benchmarks score and the flaws audits have found in them, and the black-box evaluation protocol sets out repeated trials on pinned versions.
Rillor's agent tests before go-live run on an organization's own tasks with black-box logging and replayable runs, and score consistency: the same decision across repeated runs, rephrasings and model versions.
Read the full report
The full report sets out each cause with its measured figures, what seeds, fingerprints and deterministic modes promise, what deterministic inference costs in open engines, and all 35 sources.