Where forecasts match outcomes

1 Question

How often does a CourtVision win forecast land close to what actually happened, and does that reliability hold once you condition on when in the game the forecast was made? This paper answers both for the two sports with graded in-game corpora, putting every model number next to a market reference number without assuming a shared bin label means the same rows.

This is a calibration and reliability exercise only: it reports whether a stated probability matches the observed frequency of the outcome, and not whether acting on either source's price would have produced a return.

2 Data and definitions

calibration_stability.json holds a cluster-bootstrap reliability exhibit built from data/cache/ingame_grade_joined/{mlb_segmented,soccer_intl_segmented}, dated 2026-09-16. For MLB it covers 27,351 graded rows across 178 games; for soccer_intl, 4,265 rows across 27 games. Each sport carries two independently built sets of 10 probability bins, one for model_prob and one for market_prob (the devigged closing reference). Each bin records n, n_games, mean_p, mean_y, gap (mean_y minus mean_p, so a positive gap means the outcome landed above the forecast), a bootstrapped 95 percent CI on the gap, and whether that CI excludes 0.

state_conditioned_calibration.json buckets the same rows by time_bucket (game phase) crossed with prob_bucket (a coarser 5-band split), again separately per source. For MLB it starts from 27,351 records across 178 files and drops 668 with no game-state field; for soccer_intl, 4,265 across 27 files less 882. Both the artifact and the site manifest now stamp it as_of 2026-09-16, where revision 1 published no date.

The drop rate is the sharpest sign of what segmentation did. On the joined corpus 26,340 of 78,986 MLB rows carried no game-state field; here 668 of 27,351 do, so the state-conditioned view now covers 26,683 of 27,351 MLB rows rather than two thirds of them.

Both artifacts skip mlb_clean, a byte-identical duplicate of the mlb corpus, so the same games are not counted twice.

3 Method

The aggregate bins are tested with a cluster bootstrap: resample game_id (not individual rows) with replacement 1,000 times, rebuild the 10-bin curve, and take the 2.5th and 97.5th percentile of the gap as the 95 percent CI. Resampling whole games matters because rows inside one game are correlated, and it matters more here than in revision 1 because the game count fell from 227 to 178.

gap = mean_y - mean_p; 95% CI from 1000 game-cluster bootstrap resamples; significant = (CI excludes 0)

A bin label is shared between model and market, but the two bins are not paired: each side's rows are grouped by that side's own probability. The MLB model's 0.9-1.0 bin holds n=1,437 across 61 games against the market's n=3,555 across 83, and MLB's late(inn7+), 0.8-1 cell holds n=2,369 for the model against n=2,666 for the market. Differing counts establish that the two selections are not identically selected; they do not establish that the two sets are disjoint.

4 Results

Aggregate reliability, MLB (27,351 rows, 178 games, as of 2026-09-16): the model's overall Brier is 0.1668 and the market's is 0.1510, a difference of 0.0158. Of the model's 10 bins, 3 have a gap CI excluding 0; of the market's 10, none do -- the one aggregate MLB statement that reads the same in both revisions.

The model's three significant MLB bins sit in the low and upper-middle of the range, not at the top. The largest is 0.2-0.3, predicting 0.2449 against an observed 0.1065 (n=1,774, 69 games, gap -0.1384, CI -0.2157 to -0.0448); next is 0.1-0.2, predicting 0.1557 against 0.0458 (n=1,900, 73 games, gap -0.1099); and 0.8-0.9 runs the other way, 0.8487 against 0.9414 (n=2,406, 81 games, gap 0.0927). The model is slightly too warm on longshot states and slightly too cool at 0.8-0.9.

Revision 1's headline for this table was the top bin, and it was an artifact of the join defect. On the joined corpus the model's 0.9-1.0 bin read 0.9541 predicted against 0.6632 observed over 4,753 rows, a gap of -0.2909 whose interval sat entirely below zero, reported as severe overconfidence where the model was most sure. On the segment-clean corpus the same bin reads 0.9516 against 0.9798 over 1,437 rows across 61 games, a gap of 0.0282 with a CI of -0.0063 to 0.0488 that straddles zero. A near-certain forecast attached to another game's outcome is the most expensive kind of mislabelled row, so the top bin absorbed the defect; on this corpus there is no detected miss in that bin at all.

Soccer_intl aggregate reliability (4,265 rows, 27 games): the model's Brier is 0.3325 against the market's 0.1952, and both sides show far more significant bins than MLB -- 8 of 9 eligible for the model, 7 of 9 for the market, one bin per side below the minimum-games floor. A significant interval is a departure that was detected, not a measure of how much data stands behind it. Both soccer Brier scores rose against revision 1 for a mechanical reason: the receipt records the draw share of scored ticks falling from 0.2615 to 0.1376, and a draw is cheap against a two-way probability. 27 games is a thin base for any sport-level verdict.

Calibration Stability

MLB and soccer_intl 10-bin reliability curves, model_prob vs market_prob, from the same game-cluster bootstrap exhibit as the table above.

Conditioning on game state narrows the question to game-phase x confidence-band cells. The n-weighted expected calibration error this artifact publishes is an all-state figure, not a late-game one: sports.mlb.model_ece_n_weighted is 0.0494 and sports.mlb.market_ece_n_weighted is 0.0397, weighted across early, mid and late cells alike, over the 26,683 MLB rows carrying a game-state field. Applying the same weighting to the published late(inn7+) cells alone gives a derived 0.0395 for the model and 0.0454 for the market, each over 7,442 rows; that pair is the n-weighted mean of the ten calibration_error values in the table below, not a field in the file.

Late-game MLB is not a sweep for either side. The model's largest late-inning miss is at 0.2-0.4, predicting 0.2620 against an observed 0.0843 over 617 rows (error 0.1778); the market's is at 0.4-0.6, 0.5213 against 0.3762 over 638 rows (0.1452). In the two best-populated bands both sides are close and small: at 0.8-1.0 the model reads 0.9101 against 0.9498 (n=2,369, error 0.0397) and the market 0.9206 against 0.9381 (n=2,666, 0.0175); at 0-0.2 the model reads 0.0784 against 0.0666 (n=2,884, 0.0118) and the market 0.0793 against 0.0477 (n=2,851, 0.0316). With no interval on any cell, the ordering between the two derived late figures is not a result.

The late, high-confidence cell was revision 1's second headline, and it was an artifact of the join defect too. On the joined corpus the model's late(inn7+), 0.8-1 cell read 0.9211 predicted against 0.6853 observed over 5,037 rows, a calibration error of 0.2357, described as the largest well-sampled miss in the exhibit. On the segment-clean corpus the cell holds 2,369 rows and reads 0.9101 against 0.9498, an error of 0.0397 -- a sixth of the withdrawn figure. The receipt singles this cell out as the largest single correction in the pass.

Soccer_intl's state-conditioned view runs the same direction at a coarser level -- all-state n-weighted calibration error of 0.3784 for the model against 0.2666 for the market -- on much less data: 882 of 4,265 rows carry no game-state field, leaving 3,383. Its worst cell has n=7 and an error of 0.7892, too little support to read as a measurement.

5 Robustness and what would falsify this

• If a later MLB corpus (more than the current 178 games) showed one of the model's three significant bins crossing zero, that bin's finding would no longer hold; the same in reverse for any market bin that started showing a significant gap.
• The two conclusions withdrawn here show what falsifies a result on this exhibit: both the -0.2909 top-bin gap and the 0.2357 late high-confidence error were measured correctly from a corpus whose labels were wrong, and both vanished when it was segmented. Every figure in this paper is exposed to the same class of failure.
• The state-conditioned late(inn7+), 0.8-1 cell has no bootstrapped CI of its own, and the aggregate 0.9-1.0 bin it lines up with is a coarser grouping, so the two are corroborating, not identical, evidence.
• Any soccer_intl claim is testable against more games: the current counts (model 8 of 9 eligible, market 7 of 9) would have to reproduce on a corpus well past 27 games before either is read as a property of the forecasts.

6 Limitations

• Every figure here is revision 2. Segmentation excluded 51,635 of 78,986 MLB ticks (65.4 percent) and 4,738 of 9,003 soccer ticks (52.6 percent); 27,076 MLB ticks (34.3 percent) were the identified label mismatches, the rest going with quarantined files and unlabelled segments. Intervals were re-estimated by game-cluster bootstrap, not widened. The moves against revision 1 are population changes, not evidence that the forecaster improved; it was not retrained.
• state_conditioned_calibration.json's calibration_error carries no confidence interval, unlike the bootstrapped gap_ci in calibration_stability.json, so late-inning miss sizes cannot be tested for significance, and the derived late-inning pair inherits that absence.
• model_prob and market_prob bins share a label but are built independently per source, so cells at the same nominal band are drawn from different row sets and a cell-by-cell subtraction is unpaired.
• 668 of 27,351 MLB rows and 882 of 4,265 soccer_intl rows carry no game-state field and are dropped from the state-conditioned view.
• The soccer_intl corpus is small (27 games aggregate; cells as low as n=7), so soccer verdicts are provisional and rest on a single corpus.
• Both artifacts are a single snapshot (2026-09-16) from one regeneration pass, so neither has been checked against a later or independent season, and both inherit whatever the segmentation rule decided.

7 How to read this on the site

The two tables live interactively in the "calibration" inspector (aggregate bins, both sports) and the "state-reliability" inspector (game-phase x band cells, both sources). The "observation-dependence" inspector explains why resampling game_id is the right unit here.

The figure comes from the calibration_stability module; the state-conditioned cells from the state_conditioned_calibration module, which the site marks partial pending a confidence-interval pass. The join-integrity finding page carries the receipt.