What the Brier score is made of

1 Question

CourtVision's in-game win-probability model returns a worse Brier score than the closing market in every sport where both sides are scored on the same resolved games. The Murphy decomposition splits that score into three terms that add back to it: reliability (a penalty for a forecast curve that departs from the 45-degree line), resolution (a bonus for separating winners from losers) and uncertainty (the outcome's own base-rate variance). It says which term carries a difference arithmetically, not what causes it. murphy_decomposition.json carries that split for MLB moneyline (27,351 rows) and international soccer moneyline (4,265 rows), and no others.

First this paper checks whether its own arithmetic closes: does reliability minus resolution plus uncertainty reproduce the directly computed Brier beside it. Only then does it read which component each sport's difference against the market sits in, and it separately reads brier_skill_scores.json to see whether the market's advantage is steady across the game or concentrated in one part of it. That second question is the one the corpus correction changed most.

2 Data and definitions

murphy_decomposition.json stores one object per sport under sports.<sport>, each with a model_prob block and a market_prob block. Each block carries n (rows scored), brier (the directly computed mean squared error against the 0/1 outcome), reliability, resolution, uncertainty, and reconstructed_brier -- the file's own reconstruction of the Brier from its 10 probability bins. Its method field states the identity as 'Brier = reliability - resolution + uncertainty'. Its corpus field now names the re-segmented tick paths, data/cache/ingame_grade_joined/{mlb_segmented,soccer_intl_segmented}.

Because both sides are scored on the identical set of resolved games, their uncertainty terms are identical by construction: 0.249895 for MLB, 0.213485 for soccer_intl. That match is an internal check, not a finding. MLB uncertainty rose from 0.248092 because the segment-clean base rate moved from 0.4563 to 0.4897, closer to the half-way point where base-rate variance is largest.

brier_skill_scores.json reports the same two sports' Brier scores again, this time by game-phase grain, alongside a climatology reference (a constant sport-level base-rate forecast) and the standard skill-score formula BSS = 1 - Brier / Brier_ref. Its honest_note field states plainly that BSS against the market is near or below 0 across sports and checkpoints, and that a strongly positive BSS against climatology is the expected result for any in-game forecast rather than an advantage.

Dates: revision 2 stamps these modules inside one regeneration pass. public/data/showcase/murphy_decomposition.json carries no generated_at, and states as_of 2026-09-16; site_manifest.json now lists the murphy_decomposition module with that same date, where revision 1 published it as null, and the curated insights/murphy_decomposition.json file, which previously gave its as_of as the literal string 'unknown', now carries it too. For the skill scores there are still three separate stamps: the showcase artifact brier_skill_scores.json carries generated_at 2026-09-16T22:17:01Z, which is when that artifact was produced; site_manifest.json publishes the module with as_of 2026-09-16; and the curated insights/brier_skill_scores.json file carries as_of 2026-09-16.

3 Method

Every number in the tables below is copied directly from the two named artifacts. Two derived quantities are computed here, and both are trivial differences of cited numbers, reported as such: remainder = brier - reconstructed_brier (the gap between the directly computed Brier and the identity's own reconstruction of it), and, in the skill-score table, the grain totals used to check whether the named grains sum to the all grain. The reliability_gap and resolution_gap figures quoted for MLB and soccer (model minus market) are copied from murphy_decomposition.json's own verdict text, not computed here.

No re-binning, re-scoring or walk-forward re-estimation was performed here, and neither was the re-segmentation that produced the revision-2 corpus: that is a separate pass, recorded in mlb-ingame-regeneration.json. This paper describes what the two rebuilt artifacts contain and checks whether the Murphy identity holds inside them.

4 Results

Table 1 gives the full decomposition for both sports and both sides, plus the remainder (brier minus reconstructed_brier) for each.

In MLB the two Brier figures nearly agree. The model's direct Brier (0.166759) minus its reconstructed Brier (0.168327) leaves a remainder of -0.001568; the market's (0.151008 against 0.151258) leaves -0.000250. Both are small and negative, so the 10-bin reconstruction sits very slightly above each directly computed Brier and tracks it closely enough to read its components. Model reliability (0.005619) is near market reliability (0.003425) -- a reliability_gap of +0.0022, copied from the file's verdict text -- while model resolution (0.087187) trails market resolution (0.102062), a resolution_gap of -0.0149. The model's Brier sits 0.015752 above the market's, and the resolution term carries almost all of that difference. Which term a difference sits in is a statement about the decomposition, not about why the two forecasts differ.

In international soccer both remainders are larger and negative: -0.034231 for the model (0.332522 against 0.366753) and -0.035153 for the market (0.195157 against 0.230310). The two sit within 0.001 of each other, so neither is specific to the model's probabilities. The artifact carries no second bin count and no per-bin residual, so nothing in it identifies what produces them. Soccer's model-side reliability (0.191982) is far larger than its resolution (0.038714), the reverse of MLB; on the market side reliability (0.088088) also exceeds resolution (0.071263), so the ordering flips on both soccer curves rather than the model curve alone. The file's verdict attributes the soccer difference mainly to reliability and the MLB difference mainly to resolution.

What revision 1 reported here, for the record: on the joined corpus the MLB remainders were +0.000518 (model) and -0.000377 (market), and the soccer remainders were -0.065642 (model) and -0.066812 (market). Those four figures are withdrawn. The MLB model remainder has changed sign and the soccer pair has roughly halved, which is what a large reduction in the scored population does to a bin-level reconstruction, not evidence that the identity was repaired.

5 Reliability curves

Brier = reliability - resolution + uncertainty; reconstructed_brier = reliability - resolution + uncertainty (evaluated on the file's 10 bins); remainder = brier - reconstructed_brier

6 Skill score by game phase (MLB)

brier_skill_scores.json breaks the same MLB corpus into three named game-phase grains -- early(inn1-3), mid(inn4-6), late(inn7+) -- alongside the all grain. Table 2 gives the Brier for model, market and climatology at each grain, plus both skill scores.

The model beats the climatology reference at every grain (+0.0807 early to +0.6209 late), the expected result for any in-game model against a constant base rate, rising through the game because a game in progress is easier to call than one that has not started. Against the market the model loses at every grain by a margin that is now close to flat: -0.1068 early, -0.1133 mid, -0.0918 late. The largest shortfall sits in the middle innings, and the late grain is the smallest of the three.

This reverses the revision-1 reading, and it is the clearest single consequence of the join defect. On the joined corpus the same field read -0.0794 early, -0.1547 mid and -0.4635 late, and revision 1 described the model's shortfall as growing sharply through the game. That late-game collapse was an artifact: late ticks are the ones whose probabilities have moved furthest toward a decided outcome, so a late tick carrying another game's label is scored as a confident call that was wrong, and the late grain absorbed most of the mislabelling. On the segment-clean corpus the late grain is where both forecasts do best in absolute terms (model 0.094548, market 0.086598) and where the gap between them is narrowest.

7 Robustness and what would falsify this

• If a wider MLB corpus (more than the current 27,351 rows) produced a materially larger remainder than the current -0.001568 (model) and -0.000250 (market), that would undercut trust in the 10-bin identity generally.
• If a finer bin scheme closed soccer's roughly -0.035 remainders on both curves, the remainders would be a property of the binning; this file carries one bin count (10), so that test cannot be run here.
• If the soccer model's reliability-above-resolution ordering (0.191982 against 0.038714) reversed once the corpus grew well past 4,265 rows, the current ordering would be a small-n artifact rather than a stable feature.
• The near-flat MLB BSS-vs-market profile would be falsified as a stable pattern if a later pull on an independently segmented corpus put one grain well clear of the others. The revision-1 profile was monotonic and steep, and that ordering did not survive segmentation.

8 Limitations

• murphy_decomposition.json covers only MLB moneyline and international-soccer moneyline; no NBA, tennis or WNBA rows exist in it, so nothing here generalizes to those sports.
• Every figure here is revision 2. Segmentation excluded 51,635 of 78,986 MLB ticks (65.4 percent) and 4,738 of 9,003 soccer ticks (52.6 percent); 27,076 MLB ticks (34.3 percent) were the identified label mismatches, the rest going with quarantined files and unlabelled segments. Intervals were re-estimated by game-cluster bootstrap, not widened. The moves against revision 1 are population changes, not evidence that the forecaster improved; it was not retrained.
• The 10-bin reconstruction leaves remainders near -0.035 on both soccer_intl curves, and the file carries no second bin count and no per-bin residual, so this paper cannot say what produces them.
• Reliability, resolution and uncertainty describe an aggregate corpus, not a paired game-by-game comparison.
• A resolution difference is an arithmetic contribution to the Brier difference, not a cause.
• The early/mid/late grains cover 26,683 of the all grain's 27,351 MLB rows; the remaining 668 stateless rows are not represented in the per-phase view, and this artifact prints no interval of its own.

9 How to read this on the site

The reliability curves behind Table 1 and the figure above are rendered in the score-decomposition inspector. The calibration inspector carries the wider reliability-bin context. The cross-sport-comparability inspector shows MLB and soccer_intl are not compared like for like: the corpora still differ by more than a factor of six (27,351 against 4,265 rows), and only these two sports carry a market-side decomposition.

Table 2's grain view sits alongside the decomposition table as a second way of looking at the same MLB difference: one by component, one by game phase. Both locate the difference; neither explains it. The join-integrity finding page holds the receipt, with before and after values for every figure this paper moved.