What a Brier Score Can and Cannot Tell You

A Brier score is the mean squared difference between a probability and a 0 or 1 outcome. For one recorded tick, take the published probability, subtract the terminal outcome, and square the result. Average those squared errors across the published population. The score is on a probability-squared scale, and lower values indicate lower average squared error within the compared population. It is useful because it grades probability forecasts without choosing a single cutoff. It is not an accuracy percentage, a count of correct calls, or a measure of how often a forecast landed exactly on the outcome.

The score is broad by design. A low Brier score can arise from well-calibrated probabilities, from forecasts that sort situations into meaningfully different probability levels, or from both. A higher score can have several causes. That is why the page pairs the published score with a ten-bin Murphy decomposition rather than treating a single value as a diagnosis. The decomposition states the accounting relationship as Brier = reliability - resolution + uncertainty. Each term answers a different descriptive question.

Reliability is the component associated with gaps between bin-level mean forecasts and observed frequencies. Smaller reliability values indicate less of that measured mismatch in the binned reconstruction. Resolution captures how much the binned outcome frequencies differ from the population base rate; it is subtracted in the accounting identity. Uncertainty is the outcome variability in the population before conditioning on the forecast bins. It is a property of the observed population, not a quality label attached to one series. None of these terms alone is a complete explanation of a system.

The published MLB model row is a concrete worked example. It reports a Brier score of 0.237684 across 78,986 rows. Its binned reliability component is 0.012701, resolution is 0.023628, and uncertainty is 0.248092. Applying the stated identity to those three binned quantities gives 0.012701 - 0.023628 + 0.248092 = 0.237165. The artifact publishes the reconstructed value as 0.237166 after its own stored precision. The page then compares that reconstruction with the separately published Brier: 0.237166 - 0.237684 = -0.000518. That signed value is the reconstruction remainder.

The remainder is not an error the page silently removes. It is displayed because the reconstruction was computed from ten probability bins, while the published Brier is calculated from the original tick-level probabilities. Binning compresses a probability path into ten groups. That compression can change the arithmetic even when every input field is present. The page does not attribute a remainder to rounding, data quality, or a hidden adjustment unless the artifact publishes such a cause. It reports the signed difference and leaves the reason unassigned.

This distinction matters most when comparing rows. In the same MLB population, the published market row has Brier 0.206653, reliability 0.006061, resolution 0.047124, uncertainty 0.248092, and reconstructed Brier 0.207030. That row has a different reconstruction remainder from the model row because its binned distribution differs. The shared uncertainty does not make the rows interchangeable, and the score difference alone does not say which internal component should be changed. The decomposition gives a vocabulary for describing the published gap without turning it into a causal story.

A Brier score also cannot answer questions the artifact does not measure. It does not prove that a particular probability bin is well calibrated; use the reliability diagram for the bin-level values and intervals. It does not establish independence among repeated within-game ticks; use the observation-dependence page for the lag-1 residual distributions. It does not turn a descriptive snapshot into a current forecast. The population, series, and artifact date remain part of the measurement.

Read the decomposition table as an audit trail. Start with the published Brier. Read reliability, resolution, and uncertainty as the binned components used in the reconstruction. Then inspect the signed remainder instead of assuming the reconstruction must equal the unbinned score. This is more cautious than a one-number summary, and it preserves the line between an observed calculation and an explanation that has not been published.

Inspect the worked example

Where to go next

• Score decomposition
• Calibration
• Observation dependence

Sources