How LLM forecasts compare with superforecasters and markets

A live LLM forecasting benchmark asks models for probabilities on questions with no known answer, scores the forecasts once the outcomes arrive and sets them beside human forecasters or market prices [12]. On ForecastBench's tournament leaderboard as updated on 10 November 2026, the top-ranked entry had a Brier Index of 69.6 and the superforecaster median 68.2, on a 0 to 100 scale where higher is better [2, 9]. Against professional forecasters and market prices, most of the other studies reviewed here find models level or behind on overall scores [18, 22, 24, 26].

Each figure below stays with its own question set, scale and date, and none is compared across studies.

What each LLM forecasting benchmark measures

Most of the studies here score binary forecasts with the Brier score, the squared error of the forecast probability against the outcome, where lower is better and always forecasting 50% scores 0.25 [12]. The Forecasting Research Institute (FRI), which runs ForecastBench, introduced the Brier Index in March 2026: (1 − √Brier score) × 100, so 50 means always forecasting 50% and 100 means perfect foresight [9]. Football studies score three outcomes, home win, draw or away win, on a scale where a uniform forecast scores 2/3 and the worst possible forecast 2 [27]. How the two common scores differ, and why the choice can reorder a ranking, is set out in Brier score and log loss for probability forecasts.

The timing of a forecast matters too. Paleka and colleagues call running a forecaster "on unresolved questions, waiting until the questions resolve, and then scoring the predictions" the gold standard [29]. In a backtest on questions that have already resolved, outcomes can leak in through training data or retrieval [29, 30]. So does the comparison group: a platform's crowd, superforecasters with "a strong track record of accurate performance" [12], professionals picked by a platform [20], or a market price [23].

ForecastBench results from 2024 to 2026

ForecastBench opens a round every two weeks with 500 questions, half from prediction platforms and half generated from time series [5]. Its human groups, 39 superforecasters and 500 members of the public, forecast in one round in July 2024 [1, 12].

The superforecasters' own index differs between snapshots, 70.6 in March and 68.2 in November, so entries are compared only within one snapshot [2, 9].

On 16 July 2026 FRI wrote that "several models are statistically indistinguishable from superforecaster-level accuracy" [8]. Its test takes as the null hypothesis that a model is as accurate as superforecasters. The top system's one-sided p-value was 0.41, and FRI read the results as "more consistent with superforecaster parity than with outperformance" [8]. On the 10 November board, the test does not find superforecasters more accurate than 36 entries from ten teams (p of 0.10 or more) and rates them likely more accurate than 40 more (p from 0.01 to 0.10), and the board has no test of whether an entry beats them [2].

What sits behind the parity finding

The human scores come from the July 2024 round, so later models are compared through a two-way fixed-effects model that adjusts for question difficulty [1]. FRI calls this "a statistical extrapolation that grows less reliable over time" [8].

Tournament entries may add tools, context, fine-tuning or ensembles, and the models FRI runs itself receive the crowd forecast on market questions [1]. In October 2025 FRI found that several models given market forecasts "simply copy them" [13]. One model's forecasts correlated 0.994 with the market's, with exact market values on 26 of 122 questions [13].

Without those aids, no model ranks above the public median [3]. The baseline board runs models "out of the box, without extra tools, context, or scaffolding" [1]. On 10 November 2026 it put superforecasters first at 67.1, the public median second at 62.1 and the top model third at 61.1, and every model was significantly less accurate than the superforecasters [3].

FRI's trend file of the same day projects overall parity on the tournament board in May 2027, with a range from June 2026 to March 2028 [7]. FRI's explore page says the trend line "reflects a simple extrapolation from past data" [6].

LLM forecasts against professional forecasters

Metaculus compares a team of ten Pro Forecasters with its top 10 bots on questions where the Pro forecasts are hidden, so bots cannot copy them [22]. Its head-to-head score is a log-score difference per question, where 0 means the two teams did equally well [22].

The bot team scored −20.03 in the second quarter of 2025, a gap with p = 0.00001 [19, 22]. In the season from 7 January to 15 April 2026 it scored −1.25, with a 95% interval from −4.87 to 2.37 and a one-sided p-value of 0.247 [22]. Nine of the ten Pros still beat every individual bot [22].

Lu's 2025 preprint compared models with ten expert forecasters hired by Metaculus on 2024 questions [18]. It reports that the experts "still significantly outperform the bots", setting the experts' median Brier score of 0.0225 against the top model's Brier score of 0.1352 [18]. The experts' mean on their 157 questions was 0.1573 [18].

LLM forecasts against market prices

Five studies set models directly against market prices. Lower is better in the table below, and the two football studies use the three-outcome scale.

In Prophet Arena, markets lagged several models far in advance and pulled ahead near resolution, as they took in breaking news faster [23]. TimeSeek, an April 2026 workshop paper on 150 Kalshi markets, found four of ten models ahead of the market price early in a market's life and all ten behind it near resolution [25]. The two World Cup studies covered the same matches with different models, timing and market prices, and reached different verdicts [26, 27].

Combining forecasts helped in some tests. An ensemble of the commercial system with the market price scored 0.106 against the market's 0.111 [24]. In 2024 Halawi and colleagues improved the human crowd's Brier score from 0.149 to 0.146 with a weighted average of crowd and system forecasts that gave the crowd four times the weight [16].

Leakage and sample size in LLM forecasting evaluations

Backtests carry the leakage risk. Paleka and colleagues list paths that include inaccurate dates on retrieved documents and model cutoffs that are "more of a guideline than a guarantee" [29]. Zhang and Stadie found in August 2026 that the usual check, comparing scores before and after a model's training cutoff, failed for four of five flagship models on ForecastBench questions they could not have memorized, because "recency mimics leakage" [30]. Tests for this in a backtest are set out in look-ahead bias in LLM backtests and how to test for it, and the validity and contamination flaws found in agent benchmarks in what agent benchmarks measure, and what they miss.

Sample size limits rankings too. Foresight Arena, a May 2026 preprint, scores an agent by its Alpha, the market price's Brier score minus the agent's [31]. It calculates that detecting a true Alpha of 0.02 takes about 350 resolved binary predictions at 80% power, under its stated assumptions, and an Alpha of 0.01 about four times as many [31]. Those figures are analytical, and the numbers in the paper's Section 6 come from a simulation [31]. In Metaculus's spring 2026 season, none of the top 10 bots scored clearly apart from the others [22].

The live results above share one design: each forecast exists before its outcome, and the score comes after. Rillor builds LLM forecasting agents for a client's own questions and data, and each forecast is stated before the outcome and scored against what happened.

Read the full report

The full report puts each study's question set, comparison group, scale and timing in one table, and adds the ForecastBench methods, model ensembles, the leakage disputes and where the sources disagree.