Held-Out Brier, Diebold-Mariano, and the Myths That Did Not Survive

1 Question

Two of CourtVision's published tennis receipts run a held-out Brier comparison alongside a Diebold-Mariano (DM) significance test. pregame_prior_cross_corpus asks whether a state-time prior fit on one tour's matches still beats a simple baseline when it is evaluated on the other tour's held-out states. ingame_surface_context asks a narrower question: does telling that same style of model which surface a match is on improve it over not telling it. A third receipt, tennis_grain_and_myths.json, tests three specific tennis beliefs against preregistered, leak-free verdicts: that momentum from the last point carries into the next game-level state, that some players have a persistent tiebreak-clutch skill, and that serving first in a tiebreak matters.

Before any number: neither tennis_showcase.json block compares its model to the devigged closing market price. pregame_prior_cross_corpus's own vs_close field reads UNPROVEN -- CALIBRATION only (held-out Brier), not a market advantage, and ingame_surface_context repeats the identical line. So reference in this paper means the simpler baseline model the artifact itself defines, not the market close: the BASE state-time sigmoid in the pregame test, the surface-blind hold prior in the in-game test. Held to that reference, one comparison favors the more complex model and the other favors the simpler one; both outcomes are reported plainly below.

2 Data and definitions

tennis_showcase.json carries two independent receipts, each pointing at its own upstream gate file. pregame_prior_cross_corpus is sourced from data/frontend/ingame/gate_tennis.json and runs a true A-to-B cross-corpus design: corpus A holds 29572 games and 40516 states, corpus B holds 11016 games and 14559 states (coverage.a_games, coverage.a_states, coverage.b_games, coverage.b_states). Its two named directions, atp_train_wta_test and wta_train_atp_test, each publish their own n_train_states and n_test_states, and this paper quotes those direction-level counts as published. They are not usable as tour labels: the pregame block puts 40516 states on the WTA test side while the in-game block puts the same 40516 on ATP, and 14559 the reverse way in both. The artifact does not reconcile that, so no count below is read as tour-specific. ingame_surface_context is a second, separate receipt inside the same file, sourced from data/frontend/ingame/surface_hold_verdict.json, and reports pooled and three-fold Brier for a surface-blind (H0) and a surface-specific (H1) hold prior, on each tour separately. The site manifest lists tennis_showcase's own as_of as null, so every number from it below is dated only as date not published.

tennis_grain_and_myths.json is a separate artifact, sourced from scripts/platformkit/analytics_showcase/out/fwd_claim_scoreboard.json, and carries its own generated_at of 2026-07-25T11:13:41Z. The site manifest separately lists an as_of of 2026-07-25T04:35:42Z for the same module id; this paper cites the artifact's own generated_at rather than the manifest value, since the two do not agree.

None of this is the MLB in-game corpus that companion papers restate (126 of 227 files held more than one game, 27076 of 78986 ticks mislabelled), re-segmented into revision 2 on 2026-09-16. Tennis's receipts point at gate_tennis.json and surface_hold_verdict.json; its myth tests draw on slam_points_2011_2015, slam_points, matches_asof_setdetail_schedule_density_atp_wta and travel_scouting_matches_atp_wta_2015_2025. No source field here names the MLB tick set, so that quarantine does not apply.

3 Method

A held-out Brier comparison fits each model on one slice and scores it only on a separate slice it never saw, so the gap reflects out-of-sample accuracy rather than a memorized fit. Both tennis_showcase.json blocks share the same functional form for their reference model: sigmoid((a+b*frac_elapsed)*state_diff), with a and b fit per corpus (pregame_prior_cross_corpus.base_model; ingame_surface_context.base_model uses the identical form for its surface-blind baseline). The pregame challenger blends a cross-corpus prior on top of that same base; the in-game challenger conditions the same style of hold-prior model on which surface a match is played on.

Brier(model) = mean over held-out states of (p_model - outcome)^2; brier_delta (pregame test) = brier_base - brier_prior as the artifact publishes it, so a positive value means the prior's Brier is the lower of the two.

A DM test does not test calibration. It takes the per-state loss differential between two forecasters and tests whether its mean differs from zero, using a standard error that accounts for many states from the same match being correlated rather than independent draws. The artifact's own caveat is explicit about why that matters: DM clustered by game_id -- per-state iid SE would over-state significance (pregame_prior_cross_corpus.caveats). A DM test can say one forecaster's average loss is reliably lower than another's; it cannot say either forecaster is well-calibrated in isolation, and it says nothing about a market price, which is not one of the two loss series here.

DM test (as the artifact's caveat defines the design): d_t = loss_reference(t) - loss_challenger(t) per state; reported statistic is dm_p, the two-tailed significance of mean(d) using a standard error clustered by game_id rather than treated as one independent draw per state.

4 Results

Held out on the opposite corpus's states, CourtVision's cross-corpus prior beats the plain state-diff baseline in both directions tested (Table 1): brier_prior comes in below brier_base every time, 0.1597 against 0.1698 on 40516 held-out states in one direction (brier_delta 0.01005, base minus prior), and 0.1432 against 0.1550 on 14559 held-out states in the other (brier_delta 0.01184). Both directions report dm_p 0.0 as published. A literal zero two-tailed p-value is not plausible, so it most likely reflects the artifact's rounding; the underlying DM statistic is not published, only dm_p.

Tennis Showcase

Published pregame cross-corpus and in-game surface-context Brier comparisons from tennis_showcase.json.

The in-game test asks a related but separate question: not whether a prior helps versus a flat baseline, but whether adding surface context to an already-fit hold-prior model helps versus leaving it surface-blind. The artifact's own verdict is REJECT: surface-specific hold prior does not beat surface-blind on either tour (ingame_surface_context.reason). Table 2 lays out the pooled and per-fold numbers behind that verdict.

On both tours, the surface-blind reference (H0) posts a lower, better Brier than the surface-specific challenger (H1) at the pooled level and in every one of the six published folds; sign_holds_ge_2of3 is false for both tours. The gap is small at the pooled level, 0.000121 on ATP and 0.002492 on WTA, but the sign is consistent: adding surface context did not help. The two tours carry very different uncertainty. On ATP the three fold dm_p values, 0.839017, 0.92113 and 0.607072, separate the two models in no fold: a consistent sign on a gap the test cannot distinguish from zero. On WTA the per-fold dm_p values, 0.115565, 0.0117 and 0.000188, reach conventional significance in two of three folds, with H1 the worse side. Over both, the artifact's verdict is REJECT.

tennis_grain_and_myths.json runs three preregistered story lines through the same discipline: state a hypothesis, test it against a leak-free local baseline, and record CONFIRMED_LOCAL, NULL_LOCAL or REPLICATED without editing the verdict afterward. Momentum survives at the smallest grain and dies at a larger one. Point-to-point momentum is CONFIRMED_LOCAL on 456383 point-level observations (effect 0.0216, corpus slam_points_2011_2015), but whether a player holds serve after breaking, the same idea one game-state larger, is NULL_LOCAL on 21208 states (effect 0.0051), and stays NULL_LOCAL on both halves of a split-half check (10522 states, effect 0.0067; 10686 states, effect 0.0036). Two tiebreak beliefs are tested the same way and both come back null: persistent tiebreak-clutch skill is NULL_LOCAL on 206 players (effect 0.0745), and a serve-order advantage in the breaker is NULL_LOCAL on 1281 points (effect 0.0035).

A null verdict here is not a missing result. Each null row cleared the same preregistered leak-free gate the confirmed rows did; the post-break-hold-rate null was specifically re-checked on two disjoint split halves to rule out a one-off fluctuation, and it held null on both. That is what a preregistered test is for: it can say no as readily as yes. The file's own confounds field calls CONFIRMED and NULL verdicts alike descriptive accuracy findings (tennis_grain_and_myths.json confounds).

Not everything in this file is null, and the two confirmations are not equally supported. Altitude suppresses serve ace rate (CONFIRMED_LOCAL, n=27463, effect -0.0083) and carries two further rows whose verdict field reads REPLICATED, on disjoint year slices (2015-2020: n=14329, effect -0.0078; 2021-2025: n=13134, effect -0.0089). Long travel lowers win probability (CONFIRMED_LOCAL, n=26950, effect -0.0699) and carries no replication row, so it is confirmed locally and nowhere else. The scoreboard that returned four nulls above does confirm elsewhere, and replicates in one of those two places.

5 Robustness and what would falsify this

• If additional folds of ingame_surface_context continued to show the same significantly-worse WTA pattern (dm_p 0.115565, 0.0117, 0.000188), that would harden the REJECT verdict now resting on three folds; a reversal on new folds would weaken it.
• The pregame cross-corpus test's dm_p is published as 0.0 in both directions; a future artifact publishing the underlying DM statistic and an unrounded p-value would let that reading be checked against a specific magnitude instead of treated as a rounding artifact.
• Neither tennis_showcase.json block has a vs_close value other than UNPROVEN; a future receipt joining either test to the devigged closing price would extend this comparison to a genuine market comparison, not available today.
• One corpus is about 2.8 times the other (40516 versus 14559 states), so the smaller side carries wider uncertainty, and which tour that is is exactly what the two blocks disagree on.
• tiebreak_skill_persistence_partial's n of 206 is a player-level count, far smaller than the other tested hypotheses'; a null on 206 players is real but a much less powered result than the 456383-point momentum test.

6 Limitations

• Neither tennis_showcase.json block compares any model to the devigged market close; vs_close is explicitly UNPROVEN in both, so nothing in this paper is a market comparison.
• The artifact publishes dm_p but not the underlying Diebold-Mariano statistic in either tennis_showcase.json block, so this paper's DM math block describes the test's construction from the artifact's own caveat text, not a reproducible statistic value.
• The pregame cross-corpus test has exactly two held-out directions, one per tour cross-trained on the other; it is not a k-fold cross-validation averaged over many random splits.
• The in-game surface REJECT rests on three folds per tour, a real preregistered verdict but a smaller fold count than some of CourtVision's other in-game screens.
• post_break_hold_rate's split-half check corroborates the same null on two halves of one corpus, not on a fully independent second corpus.
• tiebreak_skill_persistence_partial (n=206 players) and tiebreak_serve_order_win_rate (n=1281 points) are measured on different unit types and different corpora, so the two null tiebreak verdicts are not directly comparable in power to each other.
• tennis_surface_transfer.json's per-player clay/hard and grass splits are a separate, purely descriptive exercise with no Brier or DM test, and are not part of the held-out comparisons reported in this paper.

7 How to read this on the site

tennis_showcase renders at /analytics/m/tennis_showcase/, with both blocks browsable in full, including every published fold. tennis_grain_and_myths renders at /analytics/m/tennis_grain_and_myths/ with all three story lines and their complete claim lists. The surface-prior Brier delta behind the in-game REJECT test is also carried as its own research row, tennis-surface-prior-brier-delta, in the site's research index.

A separate paper, surface-transfer-and-form-stability, covers tennis's descriptive per-player clay/hard and grass splits from tennis_surface_transfer.json, which uses neither a Brier score nor a DM test itself.