evidence / Calibration & market
Decomposing my own Brier gap vs the market -- and publishing where the model is weakest
The most transparent sports forecaster you can audit -- every prediction pre-registered, every number gated, including the ones that refuted me. The single truth-source for any figure below is docs/JOB_EVIDENCE_PACKET.md. Everything on this page is edge_claimed=false: calibration diagnostics, not a betting signal. No dollar/ROI claim appears anywhere.
strongest single receipt
the claim
I don't just report that the market beats my in-game model -- I take my own Brier gap apart into its named components and publish the ranked list of exactly which game states my model handles worst. Two instruments do this:
1. A Murphy decomposition splits the Brier gap into reliability (miscalibration -- fixable by recalibration) and resolution (information -- the market simply resolves more state into its price). This answers why the market is ahead. 2. A state-conditioned calibration map buckets every graded in-game prediction by probability band x game-state time bucket and ranks the worst buckets. This answers where the model is weakest, as an improvement backlog.
Both run leak-free on row-level joined grade corpora, both have a --check self-check, and both write their numbers to committed JSON + a PNG.
cited artifacts
| committed artifact |
|---|
| scripts/platformkit/analytics_showcase/out/murphy_decomposition.json |
| scripts/platformkit/analytics_showcase/out/state_conditioned_calibration.json |
why this matters
Decomposing the gap into reliability vs resolution says whether the market's lead is my fault (recalibrate) or its information advantage (unavoidable given my snapshot freshness); the ranked backlog then names the exact game states where my model is worst, small-n flags and correlated-sample caveats included. The hire signal is building the diagnostic that locates the loss and writing it down instead of reporting the one number that looks good.
reproduce
# Instrument 1 -- Murphy decomposition (writes out/murphy_decomposition.json + PNG)
python -m scripts.platformkit.analytics_showcase.murphy_decomposition
python -m scripts.platformkit.analytics_showcase.murphy_decomposition --check
# Instrument 2 -- state-conditioned calibration (writes out/state_conditioned_calibration.json + PNG)
python -m scripts.platformkit.analytics_showcase.state_conditioned_calibration
python -m scripts.platformkit.analytics_showcase.state_conditioned_calibration --checkThis site reports calibration and sharpness only, never a dollar edge, ROI, or bankroll result. An honest null is a success.