How to Read a Reliability Diagram

A reliability diagram is a compact audit of whether published probabilities mean what they appear to mean. It does not show individual games. It groups many recorded probability ticks into probability ranges, then asks two separate questions within each range: what was the average forecast, and how often did the terminal outcome occur? The horizontal position is the first number. The vertical position is the second. A point on the diagonal has equal values, so the average forecast and observed frequency agree for that bin. A point above the diagonal means the observed frequency was higher than the mean forecast. A point below it means the observed frequency was lower.

The page uses ten published ranges, from 0% to 10% through 90% to 100%. The range is an inclusion rule, not the displayed forecast. A point in the 0% to 10% range can sit at 5.12%, because 5.12% is the mean of the ticks assigned to that range. Its observed frequency can be 23.31%, because that is the terminal-outcome rate among those ticks. Reading the point as a single prediction would be a category error. It is a summary of a population of ticks.

The page also shows the signed gap in percentage points. It is observed frequency minus mean forecast, not a relative percentage change. In the published MLB model 0% to 10% bin, 23.31% minus 5.12% is +18.19 pp. The pp suffix matters. It says the difference is measured on the probability scale: 0.1819, expressed as 18.19 percentage points. It does not say that the observed frequency was 18.19 percent higher in a relative sense. Keeping those languages separate makes comparisons across bins legible.

That MLB example has 5,127 ticks from 133 games. The two counts are deliberately both visible. A tick is one recorded point on a probability path. A game can contribute many ticks, and nearby ticks from the same game commonly resemble one another. So 5,127 ticks describe the plotted population, while 133 games are the clustering units behind the uncertainty calculation. Neither number replaces the other. The chart is not claiming 5,127 independent games, and it does not turn the tick total into an invented independent-sample count.

The uncertainty bars make that distinction operational. The artifact records 1,000 bootstrap resamples, clustered by game_id, with a 2.5% to 97.5% interval. Resampling games as units preserves the fact that all ticks from a selected game travel together into that resample. The published MLB example has an observed-frequency interval from 15.49% to 32.39% and a gap interval from 10.40 pp to 27.38 pp. Those intervals describe variation under the stated resampling procedure; they are not a second estimate of the point itself.

Circle size is also a reading aid rather than a claim about independence. Larger circles correspond to more ticks in the selected view, and a hollow circle marks a bin the source flags as low n. The stated floor is at least five games per bin. That floor says which bins were eligible for display. It does not make every eligible bin equally precise, and it does not erase within-game dependence. Read the confidence interval, the game count, and the low-n flag together before assigning much weight to a point.

A useful pass through the page is therefore mechanical. Select a sport and series. Read the bin range, then the mean forecast and observed frequency. Subtract in the displayed direction to understand the pp gap. Check ticks and games separately. Finally, inspect the interval and low-n status before comparing that point with another one. The reliability diagram is not a verdict on every component of a forecasting system. It is a direct display of published agreement, or disagreement, between bin-level forecasts and outcomes.

The diagram also separates model and market series instead of combining them into one curve. Those are different published populations with their own ticks, game coverage, and bins. A reader should compare like with like: model against model across ranges, market against market across ranges, then the two series with their own support in view. The page's table keeps every published bin available precisely because a smooth line can hide the count and interval that give a plotted point its context.

Inspect the worked example

Where to go next

• Calibration
• Score decomposition
• Observation dependence

Sources