Demo -- read-only snapshot of the live system as of 2026-07-16T01:45:31Z; no live data, paper units only.
Skip to main content

evidence / Calibration & market

Decomposing my own Brier gap vs the market -- and publishing where the model is weakest

The most transparent sports forecaster you can audit -- every prediction pre-registered, every number gated, including the ones that refuted me. The single truth-source for any figure below is docs/JOB_EVIDENCE_PACKET.md. Everything on this page is edge_claimed=false: calibration diagnostics, not a betting signal. No dollar/ROI claim appears anywhere.

strongest single receipt

mlb: model Brier 0.2377 is worse than market 0.2067 (gap=+0.0310); driven mainly by resolution (information, not fixable by recalibration) (reliability_gap=+0.0066, resolution_gap=-0.0235). soccer_intl: model Brier 0.2279 is worse than m...

the claim

I don't just report that the market beats my in-game model -- I take my own Brier gap apart into its named components and publish the ranked list of exactly which game states my model handles worst. Two instruments do this:

1. A Murphy decomposition splits the Brier gap into reliability (miscalibration -- fixable by recalibration) and resolution (information -- the market simply resolves more state into its price). This answers why the market is ahead. 2. A state-conditioned calibration map buckets every graded in-game prediction by probability band x game-state time bucket and ranks the worst buckets. This answers where the model is weakest, as an improvement backlog.

Both run leak-free on row-level joined grade corpora, both have a --check self-check, and both write their numbers to committed JSON + a PNG.


cited artifacts

committed artifact
scripts/platformkit/analytics_showcase/out/murphy_decomposition.json
scripts/platformkit/analytics_showcase/out/state_conditioned_calibration.json

why this matters

Decomposing the gap into reliability vs resolution says whether the market's lead is my fault (recalibrate) or its information advantage (unavoidable given my snapshot freshness); the ranked backlog then names the exact game states where my model is worst, small-n flags and correlated-sample caveats included. The hire signal is building the diagnostic that locates the loss and writing it down instead of reporting the one number that looks good.


reproduce

reproduce on a fresh clone
# Instrument 1 -- Murphy decomposition (writes out/murphy_decomposition.json + PNG)
python -m scripts.platformkit.analytics_showcase.murphy_decomposition
python -m scripts.platformkit.analytics_showcase.murphy_decomposition --check

# Instrument 2 -- state-conditioned calibration (writes out/state_conditioned_calibration.json + PNG)
python -m scripts.platformkit.analytics_showcase.state_conditioned_calibration
python -m scripts.platformkit.analytics_showcase.state_conditioned_calibration --check
no edge claimed

This site reports calibration and sharpness only, never a dollar edge, ROI, or bankroll result. An honest null is a success.