Does calibration hold up over time and across market types?

1 Question

Calibration either holds up or does not as the underlying rows change -- by calendar month, or by market type. This paper asks two narrow questions with the artifacts on hand: does CourtVision's monthly calibration error move in a consistent direction across the only two calendar months available (2026-06 and 2026-07), for MLB and international soccer (soccer_intl); and does calibration differ by market type in a way that is visible before any scoring happens, because one of those market types cannot be scored at all.

This is a descriptive reliability exercise. It reports whether stated probabilities track observed outcomes, split by month and by market type, against the devigged market close as the reference. It does not measure whether acting on either source's price would have produced a return, and it does not treat a two-month split as a real time series.

2 Data and definitions

calibration_over_time.json holds one cell per sport per calendar-month window. Only two sports and two months are present: mlb has 2026-06 (n=5,030) and 2026-07 (n=22,321); soccer_intl has 2026-06 (n=1,213) and 2026-07 (n=3,052). Each cell reports model_brier, market_brier, model_ece and market_ece for that sport-month, with a single shared n for both sides. The rebuilt artifact now carries its own as_of (2026-09-16) and names its corpus, the segmented in-game cache; the withdrawn revision published no as_of at all, so the only dating available then was the window label itself.

calibration_by_market_type.json splits the same corpus by market type instead of by month: mlb_moneyline (in-game moneyline, game winner), soccer_match (in-game match result, 2-way as graded), and mlb_total (in-game over/under run total, rescued Kalshi KXMLBTOTAL quote-tracks). Each labeled type needs at least 30 usable rows (min_n=30) before a Brier or ECE is computed, with ECE built from 10 probability bins; mlb_clean_moneyline is skipped as a byte-identical duplicate of mlb_moneyline. Only mlb_moneyline and soccer_match carry a has_outcome flag of true; mlb_total's has_outcome is false because its rescued quote-ticks were never joined to a settlement, so brier is null by design rather than computed from an invented label. Those quote-tracks are also the one part of this paper untouched by the regeneration: with no label to mis-assign, its 2,113 rows, 21 quote-tracks and 5 files read the same before and after. This artifact is dated 2026-09-16.

calibration_stability.json supplies the aggregate context: a season-long cluster-bootstrap reliability exhibit, as of 2026-09-16, covering the same two sports in full -- mlb (n_rows=27,351, n_games=178) and soccer_intl (n_rows=4,265, n_games=27, flagged low_power). For each sport it reports a model-side and a reference-side Brier, plus how many probability bins have a bootstrapped gap confidence interval that excludes zero (n_significant_bins) out of the bins with enough games to be eligible: mlb model 3 of 10 eligible bins, mlb reference 0 of 10; soccer_intl model 8 of 9, soccer_intl reference 7 of 9. Soccer loses one bin on each curve to the five-game floor, which is why its denominator is 9.

3 Method

Within one cell -- one sport-month in calibration_over_time.json, or one market type in calibration_by_market_type.json -- the model score and the reference score share a single reported n, which means both sides are scored over the same underlying rows. That makes a model-versus-reference comparison inside a single month or market type a paired, same-row comparison: both numbers were computed from the identical set of graded observations.

Comparing across cells is a different matter. June rows and July rows are different games; no shared game_id links a June observation to a July one, so a month-over-month change in ECE or Brier is an unpaired comparison between two non-overlapping samples, not a repeated measurement. The same applies across market types: mlb_moneyline and soccer_match rows are different sports and different games, so their Briers sit side by side rather than in any paired relation. Neither artifact publishes a confidence interval on a month-to-month or type-to-type difference, unlike calibration_stability.json's bootstrapped gap_ci on its bins, so only point estimates can be reported here.

gap = model_ece - market_ece (or model_brier - market_brier); no CI is published on this gap across months or market types

Comparing revision 1 to revision 2 is not a comparison either. Revision 2 is what remains after every tick whose outcome label belonged to another game was removed, so a lower Brier or calibration error after segmentation says the earlier measurement was wrong, not that the forecaster got better. Every before-and-after pair below reads that way.

The three artifacts are not three independent samples but three cuts of one corpus, and that check survives the rebuild. The MLB monthly counts sum to the MLB aggregate (5,030 + 22,321 = 27,351), which is both calibration_stability.json's sports.mlb.n_rows and market_types.mlb_moneyline.n_rows; the soccer_intl months do the same (1,213 + 3,052 = 4,265). File counts agree too: mlb_moneyline's n_files (178) equals sports.mlb.n_games and soccer_match's (27) equals sports.soccer_intl.n_games. And the market-type Briers match the aggregate Briers to six decimals, 0.166759 and 0.151008 for MLB, 0.332522 and 0.195157 for soccer.

4 Results

MLB's two months move together in level but apart in distance. The model's ECE fell from 0.1265 in June to 0.075 in July while the reference quote's fell from 0.121 to 0.0439, so the difference between them widened from 0.0055 to 0.0311. Revision 1 read this the other way, reporting a gap that narrowed from 0.0221 to 0.0199, and that reading depended on the join defect: the mislabelled ticks were not spread evenly across the two months, so removing them changed which month looked worse. Under revision 2 the honest statement is that both sides are sharper in July than in June, that the reference quote is sharper than the model in both months, and that its July advantage is the larger of the two. The Brier gap says the same more gently, 0.0095 in June against 0.0172 in July. Both gaps are trivial subtractions of two cited fields, not separately published numbers, and neither carries an interval.

Soccer_intl moved one way on both sides at once: model ECE rose from 0.3644 to 0.4243 and reference ECE from 0.2596 to 0.2952, so the distance widened from 0.1048 to 0.1291; the Brier gap barely moved, 0.1318 in June (0.3273 against 0.1955) to 0.1396 in July (0.3346 against 0.195). The direction is the one revision 1 reported, but the sample changed shape as well as size: soccer's June window is now the thinner of the two (n=1,213 against July's n=3,052), where in the withdrawn revision June was the larger. Any claim that soccer degraded across the two months now rests on the thinner month coming first.

Calibration Over Time

MLB and soccer_intl model and reference-quote calibration error and Brier, June and July 2026, from the revision-2 calibration_over_time.json.

For the two market types that can be scored, the model trails the reference quote by 0.015752 Brier on mlb_moneyline and 0.137365 on soccer_match, both read directly from brier_gap_model_minus_market. Both moved under the rebuild, in opposite directions: the MLB gap halved (0.031031 to 0.015752) and the soccer gap widened (0.08516 to 0.137365). The MLB move is the cleaner of the two -- the withdrawn number was inflated by ticks scored against another game's result. The soccer move has a mechanical cause recorded in the regeneration receipt: the draw share of scored soccer ticks fell from 0.2615 to 0.1376, and a draw scored against a two-way probability is expensive, so both sides post a higher Brier on the segmented sample (model 0.227887 to 0.332522, reference 0.142726 to 0.195157). The gap is the like-for-like line, and it widened.

Because these n, file-count and Brier values are identical to calibration_stability.json's whole-corpus numbers, the market-type table is not an independently sampled comparison for moneyline or match-result markets; it is the same season-long comparison relabeled. The only genuinely new information is the mlb_total row: 2,113 rows from 21 quote-tracks across 5 files with no resolved outcome, so instead of a Brier the artifact reports a model-versus-reference divergence (median absolute difference in probability) of 0.116128, mean 0.124782, p90 0.224736.

5 Robustness and what would falsify this

• A third calendar month for either sport would turn the current two-point comparison into a real trend check; until then, MLB's widening ECE gap (0.0055 to 0.0311) and soccer's (0.1048 to 0.1291) are each consistent with a genuine directional move or a single-month swing.
• The revision-1 finding that MLB's model-to-reference gap was narrowing month over month did not survive the rebuild; it depended on mislabelled ticks distributed unevenly across June and July. Any future claim of a narrowing gap has to be established on the segment-clean corpus rather than carried forward.
• If mlb_total's quote-tracks were ever joined to a settlement and scored, the resulting Brier could show the model closer, the reference closer, or a tie -- the disclosed divergence (median 0.116128) measures disagreement, not which side is closer to the outcome.
• The exact match between calibration_by_market_type.json's scored rows and calibration_stability.json's whole-sport aggregates is a check that passed both before and after the rebuild: had the two artifacts been built from different snapshots, the row counts, file counts and Briers would not agree.
• If a future rebuild of calibration_over_time.json added rows that do not sum to the aggregate n_rows in calibration_stability.json, the monthly view would be drawing from a different corpus; as of this snapshot the four windows account for the full aggregate.

6 Limitations

• Only two calendar months exist per sport in calibration_over_time.json; a two-point series cannot establish whether either sport's movement is a stable trend or a single-month swing, and no confidence interval is published on the change.
• The corpus behind every scored number here lost 49 of 227 MLB files and 24 of 51 soccer files to the segmentation, so revision 2 is a cleaner but much smaller sample; none of these point estimates should be compared to the withdrawn ones as if the two measured the same population.
• The mlb_moneyline and soccer_match rows are numerically identical, in n and Brier, to the aggregates in calibration_stability.json, so this market-type view adds no independent sample beyond that aggregate.
• mlb_total's divergence (median 0.116128 over 2,113 rows) measures disagreement between the two sides, not accuracy; with no resolved outcome it cannot say which side tracks what actually happened.
• Soccer_intl's June window (n=1,213) is the thinnest scored cell in either artifact and is smaller than its own July window (3,052), so the soccer comparison rests on an uneven pair of windows.
• These are calibration and reliability measurements only; nothing here establishes that acting on either source's probability would have produced a positive return.

7 How to read this on the site

The aggregate reliability bins behind calibration_stability.json are published in the "calibration" inspector with the same bootstrapped intervals cited above. A more granular cut -- game phase crossed with confidence band rather than month or market type -- is published in the "state-reliability" inspector and is the subject of the companion paper "calibration-reliability-by-sport-and-state", which draws on state_conditioned_calibration.json, likewise rebuilt on 2026-09-16.

The full account of the joined corpus, the re-segmentation and what each of the thirteen rebuilt artifacts moved by is published as the "ingame-join-integrity" finding, linked below. Five timing artifacts were not rebuilt in that pass and remain under review; none is cited here.

calibration_by_market_type is marked a partial-status exhibit on the showcase site because one of its three rows is disclosed as unscorable -- that partial status is the point of the module, not a defect.