How accurate is the reference forecast?

1 Question

CourtVision's calibration papers usually grade a model against the devigged closing price as reference. This one turns the lens on the reference itself and asks three narrower questions, each answered from one published artifact. First, when more than one source publishes a closing price for the same games, how much do their devigged Brier scores actually differ, and can this site's own data say whether that difference is more than noise. Second, does the closing price show the textbook favorite-longshot pattern -- realized win rates running below the priced probability for heavy favorites and above it for longshots -- and in which sport. Third, does accuracy improve as a price gets closer to the event start, which is the practical justification for treating the close as the reference used everywhere else on the site.

This is a descriptive exercise in the reference forecast's own reliability, not a claim about the CourtVision model. Nothing here compares the model to these sources; that comparison lives in the calibration and state-reliability inspectors and their companion papers. No number in this paper is computed beyond a same-artifact difference of two cited figures.

2 Data and definitions

Three showcase artifacts back this paper. bookmaker_accuracy.json (generated 2026-07-25) compares three named sources -- Pinnacle, MarketAvg, and Bet365 -- by proportional-devig Brier score, on the subset of games every compared source quoted: n=33,664 for tennis match winner, n=15,112 for soccer over/under 2.5 goals. market_favorite_longshot.json (generated 2026-07-25) buckets one closing price per sport by implied favorite probability: a source labeled 'Pinnacle (close)' for tennis (n_total=33,713) and one labeled 'closing line' for MLB moneyline (n_total=27,983), each bucket carrying a 95 percent Wilson interval around its realized rate. micro_closing_decay.json compares Brier at four horizons before start (T-24h, T-6h, T-1h, close) against a settled ESPN-keyed outcome, for the two sports it could join to that outcome table: WNBA (30 games, captured 2026-07-04 to 2026-07-17) and international soccer, labeled soccer_intl (13 games, captured 2026-06-19 to 2026-07-17); the site manifest lists this module's as_of as null, so the per-sport as_of fields it does publish (both 2026-07-17) are the only dates available.

All three artifacts read published pregame or pre-start closing-price data -- committed odds parquets for the first two, a scraped line-history feed of consensus-median snapshots for the third -- not the MLB in-game tick corpus the join-integrity finding flagged as partly mislabelled (126 of 227 files) and re-segmented into revision 2 on 2026-09-16. That quarantine does not apply here, and is moot for two of the three artifacts: market_favorite_longshot.json has no soccer row at all, and micro_closing_decay.json's own not_joinable.mlb field says MLB is absent for an unrelated reason -- its line-history rows carry ESPN/odds-api ids while games.parquet uses date-team keys, so zero rows join, not that any tick is mislabeled.

One naming mismatch is worth flagging before the numbers. bookmaker_accuracy.json's non-tennis sport is keyed 'soccer' and scores an over/under 2.5-goals market (n_shared=15,112), while micro_closing_decay.json's comparable sport is keyed 'soccer_intl' and scores a home-win market (n_games_joined=13). These are different market types read from different observation windows. This paper reports each artifact's own numbers under its own label and does not pool them into one soccer figure.

3 Method

bookmaker_accuracy.json defines its devig as a proportional normalization of a two-way decimal-odds pair into a probability:

p_A = (1/a) / ((1/a) + (1/b)), for decimal odds a on side A and b on side B

market_favorite_longshot.json applies the same proportional devig to a two-way closing price and then labels whichever side carries the larger implied probability the favorite:

favorite_prob = max(q, 1 - q), where q is the devigged probability of one side

Comparing two sources' accuracy is only meaningful when both are scored on the same games -- otherwise a gap could just reflect one source covering an easier slate. bookmaker_accuracy.json already restricts every row to the shared-game subset all compared sources quoted, so Table 1 below is a paired, same-game comparison in that sense. What is missing is a second layer: a confidence interval or a formal test on the gap between two sources' Brier scores. The artifact reports point estimates only, so this paper marks every row in Table 1's distinguishability column 'not published' rather than inferring significance from an eyeballed gap size.

market_favorite_longshot.json's Wilson intervals are a different kind of interval: each one bounds a single band's own realized rate against its own implied probability, band by band, not a comparison between two sources or two sports. A favorite-longshot pattern, if present, should show the gap (realized minus implied) trending in one consistent direction as the band midpoint rises -- longshots at the low end of the scale outperforming their price, favorites at the high end underperforming theirs, or the reverse; a flat, near-zero gap with no consistent direction across bands is the no-pattern case.

4 Results

Source comparison. Table 1 lists all six source-sport-market rows in bookmaker_accuracy.json. On tennis match winner (n=33,664 for every source), Pinnacle's Brier (0.1975) sits marginally below MarketAvg's (0.1977) and Bet365's (0.1980); on soccer over/under 2.5 goals (n=15,112), all three sit within 0.0001 of each other (0.2395, 0.2395, 0.2396). The artifact's own headline grades the tennis gap 'barely sharpest' and calls the soccer result a dead heat. This paper reports similar point estimates instead: no confidence interval on any pairwise gap is published, so a dead heat is not established here.

The artifact's confounds add that accuracy is not price leadership, and that the devig method shifts absolute Brier levels without changing the ranking.

Favorite-longshot shape. Table 2 gives tennis's five implied-probability bands from market_favorite_longshot.json. The gap (realized rate minus implied probability) is negative in the lowest band (-0.0026 at 0.50-0.60) and turns positive and grows through every higher band, reaching +0.0176 at 0.90-1.001 (n=1,850); the artifact's own verdict calls this a 'mild monotone favorite-longshot bias.' MLB moneyline does not show the same shape: across its four bands the gap runs +0.0032, +0.0055, -0.0078, then +0.0161, with no consistent sign as the band rises, and the artifact grades it 'essentially efficient -- no systematic bias.' Those four bands hold 27,976 games against an n_total of 27,983, so seven games sit in the total and in no displayed band; the artifact does not say why. Tennis's five bands sum to its n_total exactly.

Read one band at a time, the rule is whether that band's 95 percent Wilson interval on the realized rate excludes its own mean implied probability. Two of tennis's five bands do: 0.80-0.90, impl 0.8452 below wilson_lo 0.8483, and 0.90-1.001, impl 0.9321 below wilson_lo 0.9388. None of MLB's four do; the closest is 0.65-0.80, where impl 0.6870 sits 0.0005 inside wilson_lo 0.6865. So tennis's gap is separated from zero only in its two highest bands, and no MLB band is -- a within-sport statement, not evidence that tennis and MLB differ, since no cross-sport test is published.

Sharpening toward the close. Table 3 gives micro_closing_decay.json's Brier at each measured horizon. WNBA's Brier falls from 0.245445 at T-6h (n=26) to 0.227515 at T-1h (n=29) to 0.228045 at close (n=29); its T-24h bucket carries n=0, so no reading exists that far out. International soccer runs the other way on the same bucket-level comparison: 0.135518 at T-24h (n=7) versus 0.137424 at close (n=13), a delta the artifact itself reports as -0.0019 and labels 'does NOT sharpen,' flagged underpowered throughout.

This is where the paired-versus-unpaired distinction matters inside a single artifact. Each wnba bucket comparison uses a different subset of games (26 at T-6h, 29 at close), so it is unpaired, not a same-game repeated measurement; wnba's close_vs_t24h_paired.n is 0, so no paired test exists for wnba at all. International soccer does have one -- the seven games carrying both a T-24h and a close reading -- and it moves the same way, much further: brier_t24h=0.135518 against brier_close=0.156807, a delta of -0.021289 (close_sharper=false). The unpaired reading, 0.135518 minus 0.137424, is -0.001906. Both are negative in the artifact's t24h-minus-close convention, so both say the close is not sharper here; they differ only in magnitude, by about elevenfold.

5 Robustness and what would falsify this

• bookmaker_accuracy.json publishes point Brier only; every 'distinguishable' cell in Table 1 is reported not published because no confidence interval or paired test on the pairwise gap is in the artifact.
• market_favorite_longshot.json's Wilson intervals bound one band's own gap, not a comparison across bands, sports, or sources; the reading here would be falsified by a rebuild showing MLB's bands turning monotone like tennis's, or tennis's two top intervals coming to contain their own implied rates.
• The soccer_intl sharpening question would be settled differently by a larger paired sample: the current paired test (n=7) agrees in sign with the unpaired bucket reading and is about eleven times its size.
• wnba's own sharpening claim rests entirely on an unpaired comparison across 26-to-29-game buckets, because its T-24h bucket and its paired-test denominator are both n=0; a wider capture window would allow a first paired test.
• bookmaker_accuracy.json's 'soccer' (over/under 2.5 goals) and micro_closing_decay.json's 'soccer_intl' (home win) come from different market types on different windows; treating them as one soccer answer would be a falsifiable overreach this paper avoids.

6 Limitations

• No confidence interval or significance test on the source-to-source Brier gap is published in bookmaker_accuracy.json, so this paper cannot say whether Pinnacle, MarketAvg, or Bet365 differ by more than noise on either market.
• The sharpening-toward-close question in micro_closing_decay.json is answered for only two sports and 43 total joined games (30 wnba, 13 soccer_intl); MLB, NBA, and tennis are absent from this instrument for join-key reasons unrelated to the close itself.
• Every soccer_intl bucket in micro_closing_decay.json is flagged underpowered (n=7 to 13), and its one paired same-game test (n=7) agrees in sign with the unpaired bucket-level comparison quoted in the artifact's own headline but is about eleven times its size.
• bookmaker_accuracy.json's 'soccer' (over/under 2.5 goals) and micro_closing_decay.json's 'soccer_intl' (home win) are different market types on different windows and are not pooled into one soccer reading here.
• Proportional devig is one normalization among several; the source artifacts note it shifts absolute Brier and probability levels without changing which source or band ranks where, so a different devig method could move every number in this paper without changing its qualitative shape.
• These are reliability measurements of the reference forecast's own closing price; nothing here scores the CourtVision model against these sources, and nothing here should be read as a statement about which source is more useful to follow.

7 How to read this on the site

The calibration inspector (inspector id calibration) carries the model-versus-market reliability bins that the companion paper calibration-over-time-and-market-type restates by month and market type; this paper's tables sit upstream of that comparison, describing the reference side alone. closing-reference-movement-measured covers a related question -- how the reference moves before it closes -- using different artifacts than the three read here.

All three artifacts publish their full per-sport and per-bucket arrays on the analytics site, including the international-soccer paired subsample and the not_joinable reasons for every excluded sport, which this paper's tables summarize but do not reproduce in full.