The Brier score and log loss both grade a probability forecast against what happened. The Brier score is the squared gap between the forecast probability and the outcome, coded 1 if the event happened and 0 if it did not [12, 24]. Log loss is the negative logarithm of the probability the forecast gave to the outcome that occurred [13, 15]. Both are strictly proper: a forecaster gets the best expected score only by reporting the probability they believe [5, 17].
The difference between the Brier score and log loss shows up in confident misses. The Brier penalty for one binary forecast is at most 1 on the 0 to 1 scale [12]. Log loss grows without limit as the probability on the outcome falls toward zero, and a forecast of 0 on an event that happens scores infinity [8, 12]. That difference can reorder a ranking. In one study of 10 forecasters on 21 questions, the Brier and log rankings had a Spearman correlation of 0.15 [19]. The choice follows the cost of a confident miss. Serafini and colleagues suggest the log score for an alarm system, where overconfidence "may put lives at risk", and the Brier score where being overconfident "is less dangerous" [20].
Before comparing published Brier scores, check the scale. Some sources report 0 to 1 and others 0 to 2, so the same binary forecasts can produce numbers a factor of two apart [13, 14, 22].
What makes a scoring rule proper
A scoring rule is proper if the forecaster maximizes the expected score by issuing the forecast they hold, and strictly proper if that maximum is unique [5]. In Murphy's proof for the Brier score, a forecaster who believes the probability is p and reports f expects a score of p(1 - p) + (f - p)², which is lowest at f = p [4].
Rules that look sensible can fail this test. The linear score, which scores a forecast by the probability it gave to the outcome, is improper [5]. In a Metaculus example, if the true chance of a six on a die is 17%, a forecast of 5% earns a better average linear score than a forecast of 17% [23]. Mean absolute error is improper for binary events too, because it rewards forecasts of 0 or 1 [26].
Propriety matters even when nobody is gaming the score. Bröcker and Smith show that an improper score can rank one forecast above another even when the data were drawn from the second, and that whether a score is proper can be settled before any data are seen [7].
One forecast through both scores
For binary forecasts f with outcomes o (1 or 0), the Brier score on the 0 to 1 scale is the mean of (f - o)² [12, 24], and log loss is the mean of -(o ln f + (1 - o) ln(1 - f)) [13, 15]. The table shows single forecasts on both scores.
| Forecast for "yes" | Outcome | Brier, 0 to 1 scale | Brier, 0 to 2 scale | Log loss (natural log) |
|---|---|---|---|---|
| 0.50 | Either | 0.250 | 0.500 | 0.693 |
| 0.70 | Yes | 0.090 | 0.180 | 0.357 |
| 0.70 | No | 0.490 | 0.980 | 1.204 |
| 0.90 | No | 0.810 | 1.620 | 2.303 |
| 0.99 | No | 0.980 | 1.960 | 4.605 |
| 0.999 | No | 0.998 | 1.996 | 6.908 |
Computed from the formulas in [12, 13, 15], matching Good Judgment Open's 0.18 example [22] and ForecastBench's 0.25 for a 50% forecast [24].
On a miss, moving from 0.70 to 0.99 doubles the Brier penalty and nearly quadruples the log loss. On Metaculus's log score, going from 99% to 99.9% gains 0.009 if the forecast is right and costs 2.3 if it is wrong [23].
Brier score vs log loss: where they disagree
Confident misses. Merkle and Steyvers compared 10 forecasters on 21 binary questions from a public forecasting site. One forecaster ranked 3rd under the Brier score and 10th under the log score. That forecaster "tended to report extreme forecasts, and the incorrect forecasts were heavily penalized under the logarithmic score" [19]. Four other forecasters moved four places. The authors note the small sample [19].
Rare events. Serafini and colleagues compared the scores on earthquake forecasts with a true probability of 0.001. Under the Brier score, a forecast of 0 is preferred to any forecast more than twice the true probability [20]. The Brier score also prefers a forecast one thousand times too small to one four times too large, and the log score prefers the reverse [20]. The Joint Working Group on Forecast Verification Research warns that the rarer the event, the easier it is to get a good Brier score without real skill [12].
Overconfidence and underconfidence. The Brier score penalizes both the same in probability space, and the log score penalizes overconfidence more [17]. If the true probability is 0.70, reporting 0.55 or 0.85 adds the same 0.0225 to the expected Brier score on the 0 to 1 scale. The expected log loss, in nats, rises by 0.047 for 0.55 and by 0.072 for 0.85 (computed from the formulas in [4, 13]).
The Brier score decomposition: reliability, resolution and uncertainty
In 1973 Murphy split the Brier score into three terms: score = reliability - resolution + uncertainty [2]. Reliability measures how far each group of identical forecasts sits from the frequency observed in that group. Resolution measures how far those group frequencies sit from the overall frequency. Uncertainty is the score of always forecasting the overall frequency [2, 12]. In Murphy's example of ten forecasts, 0.136 - 0.330 + 0.480 = 0.286 on the 0 to 2 scale [2]. Proper scores in general, the log score included, decompose the same way [10].
The split shows why a low Brier score does not prove calibration. scikit-learn's documentation notes that a lower Brier loss "does not necessarily mean a better calibrated model", since a worse calibrated model can have much more discriminatory power [13]. The terms also catch a complex forecast that loses to a simple one. For 24-hour rain forecasts at Niamey, Niger, from July to September 2016, a 52-member ensemble scored 0.266 on the 0 to 1 scale, against 0.234 for a climatology-based reference forecast [11]. Its miscalibration term alone was 0.066 [11].
How to measure forecast calibration
A reliability diagram plots the observed frequency against the forecast probability [12, 16]. Points below the diagonal mean the probabilities were too high [12]. With the usual binning, Dimitriadis, Gneiting and Jordan show the picture can change drastically between 9, 10 and 11 bins [11]. Calibration alone is not the goal. Gneiting and colleagues define the aim as "maximizing the sharpness of the predictive distributions subject to calibration" [6]. Always forecasting the long-run frequency is calibrated by construction and has no sharpness [5, 12].
Two Brier scales, daily averages and skill scores
Brier defined his score in 1950 as a sum over every category, with a range of 0 to 2 [1]. Good Judgment Open sums over every answer option, so a 70% forecast on an event that happens scores 0.18 on the same 0 to 2 scale [22]. It scores each active day, carrying a forecast forward until it is updated, and averages the daily scores [22]. ForecastBench uses (f - o)² on the 0 to 1 scale [24]. scikit-learn's brier_score_loss halves the sum for binary targets by default (scale_by_half="auto"), giving 0 to 1, and reports 0 to 2 with scale_by_half=False [14].
Brier skill score
A raw score means little without a reference. A skill score compares a forecast with a reference such as climatology or persistence. It equals 1 for a perfect forecast, 0 for the reference and less than 0 for a forecast worse than the reference [5, 12]. The Brier skill score is "not strictly proper" and unstable on small samples [12], though Murphy showed that a skill score defined on a whole collection of forecasts is approximately strictly proper once there are 100 or more forecasts [3]. On a 200-question ForecastBench subset scored on the 0 to 1 scale, the superforecasters' median forecast scored 0.096, the public's median 0.121 and the top model 0.122 [24]. Agent evaluation metrics covers what other model benchmark scores measure.
Choosing and reporting a score
- Use log loss where a confident miss should cost a lot, and handle probabilities near 0 and 1 with care. scikit-learn clips them at machine precision [15], Metaculus limits binary forecasts to 0.1% to 99.9% [23], and truncating the log score at a fixed floor makes it improper [27].
- State the Brier scale, the log base and the reference forecast next to every number.
- Put error bounds on score differences, especially for rare events [12]. The black-box evaluation protocol covers error bars.
- Score only forecasts stated before the outcome was known, as ForecastBench does to avoid data leakage [24].
Rillor's forecasting and market intelligence work checks stated probabilities against observed frequencies and compares each forecast with simple baselines, such as no change or the historical rate. It builds forecasting systems on a client's own questions and data, scored the way it scores its own.
Read the full report
The full report adds the formal definitions, the skill-score formulas, how forecasting platforms average and rescale scores, scores for quantile and interval forecasts and all 28 sources.