Skip to main content
Read-only snapshot: model data as of 2026-07-16; paper desk record through 2026-10-22. No live data; paper units only.
CourtVision
Ask Scout

Calibration against the market

Verdict. Pregame vs the Devigged close the closing line with the vig removed, i.e. the market's fair probability at the final pre-game price; the comparison forecaster for every pregame-vs-close row (see Devig / Shin devig and Closing line).: MLB moneyline TRAILS_CLOSE; Soccer O/U 2.5 TRAILS_CLOSE; NBA moneyline cannot tell (interval includes zero; underpowered). In play: MLB in-play win probability BEHIND (vs the Kalshi in-play mid); International soccer in-play win probability BEHIND (vs the Kalshi in-play mid). Every row below is scored by Brier score mean squared error of probabilistic forecasts (lower = sharper). (lower is better) and shows its n, 95% interval, as-of date and receipt.

Status: TRAILS_CLOSE on 2 of 4 pregame rows, BEHIND on 2 of 2 in-play rows ยท as of 2026-09-24

Verdict table

Brier score mean squared error of probabilistic forecasts (lower = sharper). is the mean squared error of the probability; lower is better. The 0.2500 baseline is a constant 50% forecast. MATCHES_CLOSE carries UNDERPOWERED where the row's interval is wider than its MDE (minimum detectable effect) the smallest Brier gap a row's 95% interval could have resolved, 1.429 x the interval half-width (DM_RECOMPUTE_2026-09-24); a row whose measured gap is below its MDE is underpowered, so MATCHES_CLOSE (underpowered) means "not resolved", not parity.: not resolved, not parity.

Model against the market reference, Brier (lower is better)
SportMarketCheckpointReferencen gamesn rowsBrier modelBrier marketBrier baselineGap (model minus market), 95% CIVerdictAs of
MLBmoneylinepregameproportional devigged close13,99213,992 (one per game)0.2429 (lower is better)0.23900.2500+0.0039 [+0.0028, +0.0051]TRAILS_CLOSE2026-09-24
SoccerO/U 2.5pregameproportional devigged close7,5587,558 (one per game)0.2465 (lower is better)0.23900.2500+0.0076 [+0.0059, +0.0092]TRAILS_CLOSE2026-09-24
NBAmoneylinepregameshin devigged close372372 (one per game)0.1735 (lower is better)0.16660.2500+0.0069 [-0.0038, +0.0175]MATCHES_CLOSE UNDERPOWERED2026-09-24
NBAmoneylinepregameproportional devigged close372372 (one per game)0.1735 (lower is better)0.16720.2500+0.0063 [-0.0042, +0.0168]MATCHES_CLOSE UNDERPOWERED2026-09-24
MLBin-play win probabilityin-play ticksKalshi in-play mid*227not published0.2377 (lower is better)0.20670.2500+0.0310 [+0.0170, +0.0454]BEHIND2026-09-14
International soccerin-play win probabilityin-play ticksKalshi in-play mid**51not published0.2279 (lower is better)0.14270.2500+0.0852 [+0.0488, +0.1270]BEHIND2026-09-14

* Measured 2026-09-14 on the 227-game corpus that later failed the join-integrity check; the 178-game segmented revision keeps the verdict BEHIND. See the join-integrity finding and the in-play latency paper.

** Measured before the join-integrity repair; not re-established after it.

Reliability diagrams with game-cluster intervals

Each point is one probability bin: mean forecast against observed frequency, with the diagonal as the calibrated reference. Intervals use 1,000 bootstrap resamples clustered by game_id (2.5% to 97.5%). Bins under the 5-game floor are hollow and marked below floor. n counts ticks; the table gives games per bin.

Data integrity

These MLB/soccer numbers are revision 2, computed on the segment-clean corpus (2026-09-16). Revision 1 values are withdrawn and kept in the regeneration receipt. Segmentation excluded 51,635 of the 78,986 MLB ticks (65.4 percent); the specifically identified label mismatches were a smaller 27,076 (34.3 percent).

  • calibration_stability
Read the full finding

Reading trail

Read first: Read the reliability finding before comparing published bins.

Next question: Where do repeated observations reduce the distinct support behind a bin?

Read the analysis: Does calibration hold up over time and across market types? (sources regenerated)

PaperSupporting sourceDoes calibration hold up over time and across market types? (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceHome and away, taken apart (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceHow accurate is the reference forecast? (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceHow many games actually sit behind a bin (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceHow the closing reference moves (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceHow to read a CourtVision paper (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceThe graveyard: what the null results teach (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceThe rotation day that was already known (sources regenerated)MLB | as_of 2026-09-16PaperSupporting sourceWhat carries across sports (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceWhat the Brier score is made of (sources regenerated)Cross-sport | as_of 2026-09-16PaperSupporting sourceWhere forecasts match outcomes (sources regenerated)Cross-sport | as_of 2026-09-16ModuleSupporting sourceCalibration StabilityCross-sport | as_of 2026-09-16AnalysisSame source fileCalibration support concentrationCross-sport | as_of 2026-09-16AnalysisSame source fileSigned calibration directionCross-sport | as_of 2026-09-16

MLB: 27,351 ticks from 178 games; ticks are not independent games.

Perfect calibrationModelMarketLow-n bin
0%0%25%25%50%50%75%75%100%100%Mean forecast probabilityObserved frequencyModelModel 5.1% forecast, 6.3% observed (interval 1.4% to 13.4%), 1.19 pp gap, 2,072 ticks from 86 gamesModel 15.6% forecast, 4.6% observed (interval 0.4% to 11.2%), -10.99 pp gap, 1,900 ticks from 73 gamesModel 24.5% forecast, 10.7% observed (interval 3.2% to 19.8%), -13.84 pp gap, 1,774 ticks from 69 gamesModel 35.8% forecast, 23.5% observed (interval 12.8% to 37.5%), -12.32 pp gap, 2,217 ticks from 80 gamesModel 45.8% forecast, 37.1% observed (interval 23.7% to 51.3%), -8.67 pp gap, 3,859 ticks from 100 gamesModel 54.3% forecast, 55.2% observed (interval 44.0% to 66.7%), 0.87 pp gap, 5,636 ticks from 114 gamesModel 64.7% forecast, 64.3% observed (interval 49.5% to 78.0%), -0.43 pp gap, 4,026 ticks from 95 gamesModel 74.6% forecast, 82.2% observed (interval 70.8% to 92.1%), 7.52 pp gap, 2,024 ticks from 70 gamesModel 84.9% forecast, 94.1% observed (interval 87.7% to 98.4%), 9.27 pp gap, 2,406 ticks from 81 gamesModel 95.2% forecast, 98.0% observed (interval 94.5% to 100.0%), 2.82 pp gap, 1,437 ticks from 61 gamesMarketMarket 4.9% forecast, 2.1% observed (interval 0.0% to 5.6%), -2.78 pp gap, 3,233 ticks from 83 gamesMarket 14.8% forecast, 12.1% observed (interval 3.8% to 24.3%), -2.69 pp gap, 2,344 ticks from 87 gamesMarket 24.8% forecast, 15.9% observed (interval 7.7% to 27.7%), -8.85 pp gap, 1,876 ticks from 90 gamesMarket 35.1% forecast, 23.7% observed (interval 13.0% to 35.3%), -11.43 pp gap, 1,526 ticks from 98 gamesMarket 45.3% forecast, 34.5% observed (interval 22.1% to 48.3%), -10.75 pp gap, 2,703 ticks from 127 gamesMarket 55.0% forecast, 49.3% observed (interval 38.8% to 60.9%), -5.66 pp gap, 4,129 ticks from 140 gamesMarket 64.9% forecast, 68.2% observed (interval 56.2% to 78.6%), 3.31 pp gap, 2,957 ticks from 117 gamesMarket 74.6% forecast, 70.4% observed (interval 56.7% to 82.7%), -4.24 pp gap, 2,579 ticks from 100 gamesMarket 85.2% forecast, 86.3% observed (interval 76.6% to 94.5%), 1.10 pp gap, 2,449 ticks from 88 gamesMarket 94.9% forecast, 97.5% observed (interval 94.3% to 99.5%), 2.53 pp gap, 3,555 ticks from 83 games

MLB / Model worked example

0% to 10% probability bin

Mean forecast 5.1% and observed frequency 6.3% produce a published 1.19 pp gap. The observed-frequency interval is 1.4% to 13.4%.

Eligible bins
10
Significant bins
3
Within noise
7
Published Brier
0.166759

MLB / Market worked example

0% to 10% probability bin

Mean forecast 4.9% and observed frequency 2.1% produce a published -2.78 pp gap. The observed-frequency interval is 0.0% to 5.6%.

Eligible bins
10
Significant bins
0
Within noise
10
Published Brier
0.151008
Published reliability bins for MLB
SeriesBin rangeMean forecastObservedGap (pp)Gap CI (pp)n ticksn gamesLow nInspect
Model0% to 10%5.1%6.3%1.19 pp-3.66 pp to 8.21 pp2,07286No
Model10% to 20%15.6%4.6%-10.99 pp-15.30 pp to -4.56 pp1,90073No
Model20% to 30%24.5%10.7%-13.84 pp-21.57 pp to -4.48 pp1,77469No
Model30% to 40%35.8%23.5%-12.32 pp-23.03 pp to 1.39 pp2,21780No
Model40% to 50%45.8%37.1%-8.67 pp-21.95 pp to 5.12 pp3,859100No
Model50% to 60%54.3%55.2%0.87 pp-10.26 pp to 12.17 pp5,636114No
Model60% to 70%64.7%64.3%-0.43 pp-14.75 pp to 13.40 pp4,02695No
Model70% to 80%74.6%82.2%7.52 pp-3.76 pp to 17.75 pp2,02470No
Model80% to 90%84.9%94.1%9.27 pp2.98 pp to 13.76 pp2,40681No
Model90% to 100%95.2%98.0%2.82 pp-0.63 pp to 4.88 pp1,43761No
Market0% to 10%4.9%2.1%-2.78 pp-4.84 pp to 0.55 pp3,23383No
Market10% to 20%14.8%12.1%-2.69 pp-11.03 pp to 9.43 pp2,34487No
Market20% to 30%24.8%15.9%-8.85 pp-17.20 pp to 2.90 pp1,87690No
Market30% to 40%35.1%23.7%-11.43 pp-22.02 pp to 0.24 pp1,52698No
Market40% to 50%45.3%34.5%-10.75 pp-23.20 pp to 3.14 pp2,703127No
Market50% to 60%55.0%49.3%-5.66 pp-16.18 pp to 5.95 pp4,129140No
Market60% to 70%64.9%68.2%3.31 pp-8.70 pp to 13.75 pp2,957117No
Market70% to 80%74.6%70.4%-4.24 pp-18.01 pp to 8.23 pp2,579100No
Market80% to 90%85.2%86.3%1.10 pp-8.35 pp to 9.11 pp2,44988No
Market90% to 100%94.9%97.5%2.53 pp-0.63 pp to 4.48 pp3,55583No

Brier decomposition

Calibration vs resolution (Murphy decomposition) Brier splits into reliability (the calibration error: how far observed frequency sits from the forecast in each bin, fixable by recalibrating) and resolution (how much the forecasts separate outcomes: information that recalibrating cannot manufacture). A market can lead on resolution while both forecasters are calibrated.: Brier = reliability - resolution + uncertainty, computed from the published bins. Within-bin spread is not published, so the binned total sits near, not on, the published Brier.

Brier decomposition by sport and series (lower Brier is better; uninformative baseline 0.2500)
SportSeriesReliability (lower is better)Resolution (higher is better)UncertaintyBinned totalPublished Brier (lower is better)n ticks
MLBmodel0.00560.08720.24990.16830.166827,351
MLBmarket0.00340.10210.24990.15120.151027,351
International soccermodel0.19200.03870.21350.36680.33254,265
International soccermarket0.08810.07130.21350.23030.19524,265

Sharpness

How forecasts spread across the probability bins. A sharper forecaster puts more forecasts near 0% and 100%; sharpness is only useful when the bins above stay calibrated.

02,8185,6360%10%20%30%40%50%60%70%80%90%MLB model: 2,072 forecasts in 0% to 10%MLB model: 1,900 forecasts in 10% to 20%MLB model: 1,774 forecasts in 20% to 30%MLB model: 2,217 forecasts in 30% to 40%MLB model: 3,859 forecasts in 40% to 50%MLB model: 5,636 forecasts in 50% to 60%MLB model: 4,026 forecasts in 60% to 70%MLB model: 2,024 forecasts in 70% to 80%MLB model: 2,406 forecasts in 80% to 90%MLB model: 1,437 forecasts in 90% to 100%MLB market: 3,233 forecasts in 0% to 10%MLB market: 2,344 forecasts in 10% to 20%MLB market: 1,876 forecasts in 20% to 30%MLB market: 1,526 forecasts in 30% to 40%MLB market: 2,703 forecasts in 40% to 50%MLB market: 4,129 forecasts in 50% to 60%MLB market: 2,957 forecasts in 60% to 70%MLB market: 2,579 forecasts in 70% to 80%MLB market: 2,449 forecasts in 80% to 90%MLB market: 3,555 forecasts in 90% to 100%
MLB. Legend: model = left bar of each pair, market = right bar. Counts are ticks, not games.
Show as table
Binmodel forecastsmarket forecasts
0% to 10%2,0723,233
10% to 20%1,9002,344
20% to 30%1,7741,876
30% to 40%2,2171,526
40% to 50%3,8592,703
50% to 60%5,6364,129
60% to 70%4,0262,957
70% to 80%2,0242,579
80% to 90%2,4062,449
90% to 100%1,4373,555
06421,2840%10%20%30%40%50%60%70%80%90%International soccer model: 1,284 forecasts in 0% to 10%International soccer model: 792 forecasts in 10% to 20%International soccer model: 669 forecasts in 20% to 30%International soccer model: 449 forecasts in 30% to 40%International soccer model: 180 forecasts in 40% to 50%International soccer model: 202 forecasts in 50% to 60%International soccer model: 136 forecasts in 60% to 70%International soccer model: 198 forecasts in 70% to 80%International soccer model: 194 forecasts in 80% to 90%International soccer model: 161 forecasts in 90% to 100%International soccer market: 603 forecasts in 0% to 10%International soccer market: 486 forecasts in 10% to 20%International soccer market: 969 forecasts in 20% to 30%International soccer market: 224 forecasts in 30% to 40%International soccer market: 405 forecasts in 40% to 50%International soccer market: 467 forecasts in 50% to 60%International soccer market: 223 forecasts in 60% to 70%International soccer market: 271 forecasts in 70% to 80%International soccer market: 474 forecasts in 80% to 90%International soccer market: 143 forecasts in 90% to 100%
International soccer. Legend: model = left bar of each pair, market = right bar. Counts are ticks, not games.
Show as table
Binmodel forecastsmarket forecasts
0% to 10%1,284603
10% to 20%792486
20% to 30%669969
30% to 40%449224
40% to 50%180405
50% to 60%202467
60% to 70%136223
70% to 80%198271
80% to 90%194474
90% to 100%161143

Coverage

Coverage of interval forecasts: NOT_TESTABLE in this snapshot. The rows above are binary probabilities; no nominal-level interval forecasts are published here, so no empirical coverage is printed.

Inspectors

Summary by market

Published Brier (lower is better) and expected calibration error, where resolved outcomes exist.

SportMarketModel Brier (lower is better)Market BrierBaseline BrierModel ECEMarket ECEn rows
MLBMLB in-game moneyline (game winner)0.16680.15100.25000.05740.049127,351
International soccersoccer in-game match result (2-way, as graded)0.33250.19520.25000.40020.28194,265
MLBMLB in-game over/under run total (Kalshi KXMLBTOTAL quote-tracks)Not scoredNot scored0.2500Not scoredNot scored2,113

Limitations

  • Sample and independence: reliability n counts ticks; ticks within a game are not independent, which is why intervals are clustered by game.
  • Scope: reliability bins cover MLB and international soccer only; the pregame verdicts come from one recompute memo dated 2026-09-24.
  • What this does not establish: a calibrated forecast that trails or matches the market is not a reason to trade against it.

Reproduce

python scripts/platformkit/analytics_showcase/check_all.py

Source path

docs/evidence/CANONICAL_FIGURES.json; webapp/public/data/showcase/calibration_stability.json; docs/evidence/pregame/DM_RECOMPUTE_2026-09-24.md

Quoted on

docs/evidence/pregame/DM_RECOMPUTE_2026-09-24.md is quoted on:

calibration_stability.json is quoted on:

Change log

2026-10-20: rebuilt to the T2 template (verdict table, decomposition, sharpness, coverage; validated palette).