How proposed signals survive testing

1 Question

CourtVision runs many more candidate signals through its validation ledger than it ever publishes as findings. This note adds no new signal and no number computed outside that ledger; it reads the ledger itself -- what was proposed, what was tested, what the verdict was at each check, and what changed later -- as the object of study. The question is not whether any one signal holds up. It is what this dated snapshot records: whether a verdict is corrected when a re-test disagrees with it, and whether the outcomes that did not confirm are kept and counted rather than dropped.

Four artifacts are described here, all reading the same ledger. fwd_claim_scoreboard.json gives the current status of every claim family in the 2026-07-11 snapshot. verdict_flip_anatomy.json gives the five families whose verdict changed as they were re-tested, and the seven retracted outright. mechanism_survival.json gives the row-level survival rate of every recorded mechanism test, and mechanism_ledger_export.json is a companion export of the same rows that counts confirmations slightly differently. That last comparison is the point: two files built from the same test runs disagree on the confirmed count by exactly one row, because they draw the bucket boundary around one verdict label differently. A ledger that reports a single confirmation percentage without saying where that boundary sits is reporting a choice, not a fact.

2 Data and definitions

fwd_claim_scoreboard.json groups the validation ledger into claim families, one row per sport-and-hypothesis pair, and lists each family's full verdict history ordered by run_ts. The file's own as_of is 2026-07-11T16:48:14Z. It holds 287 individual test-run rows (n_ledger_rows) across 259 distinct families (summary.n_families); a family with more than one recorded run has n_entries greater than one, and its current_status is whatever verdict its latest history entry carries. A family is flagged flipped when its history contains more than one distinct status.

verdict_flip_anatomy.json is a companion file, generated at 2026-07-25T11:13:42Z, built over the same ledger and focused on two things: the five families the scoreboard flags as flipped, with every step of their verdict sequence (flips[].steps[]), and the seven families whose current status is retracted, each with a short account of what ended it (retracted[].what_killed_it). Its own dating_coverage field records that of 287 total history rows, only 145 carry a run_ts; the remaining 142 are undated and are kept in source-file order rather than a confirmed chronological order.

mechanism_survival.json and mechanism_ledger_export.json both describe the same 287 ledger rows, but at the row level rather than the family level -- one row per recorded test, not deduplicated to one row per sport-and-hypothesis pair -- and neither file publishes an as_of date (date not published for both). Each defines a three-way verdict taxonomy: a confirmed bucket (CONFIRMED_LOCAL, CONFIRMED_LOCAL_incl_2026_OOS, REPLICATED, and, in mechanism_ledger_export.json only, ARTIFACT_CONFIRMED), a not_testable bucket (NOT_REPLICABLE_NO_CORPUS, NOT_TESTABLE), and everything else. mechanism_survival.json additionally sorts every row into one of seven keyword-matched categories: rest_fatigue, travel, streak_momentum, matchup, weather_park, official_referee, and a situational catch-all for rows matching none of the others.

3 Method

A family's current status is a single label, taken from the last entry in its own history array; when run_ts is present the history is ordered by run_ts, and when it is absent (142 of 287 rows) the order is whatever order the source ledger already held. This matters for reading a verdict sequence: for four of the five flipped families below, no step carries a run_ts, so 'first' and 'last' describe file order, not a dated timeline. The fifth family's three steps do carry a timestamp, but it is the identical timestamp on all three, so it establishes only that the steps were logged together, not how much time separated them.

mechanism_survival.json's survival_rate is n_confirmed divided by n_testable, not by n, the raw row count, so not_testable rows are removed from both the numerator and the denominator before the rate is taken. Overall: n=287, n_testable=256, n_confirmed=129, survival_rate=0.5039. mechanism_ledger_export.json counts the same 287 rows into overall_counts (confirmed=130, null=126, not_testable=31) using a bucket definition that differs from mechanism_survival.json's on exactly one verdict label: ARTIFACT_CONFIRMED sits inside the confirmed bucket in mechanism_ledger_export.json's verdict_buckets, and inside the everything-else bucket in mechanism_survival.json's verdict_taxonomy. The gap between 129 and 130 confirmed rows is that single label's reclassification (130 minus 129 equals 1 row), not a new measurement.

4 Results

Table 1 reads the snapshot's own count of where each of the 259 claim families stands, by sport.

121 families are currently verified. 134 families are null, not testable or retracted (summary.by_status: null 99, not_testable 28, retracted 7), and four more are provisional, so 138 of the 259 families carry a status other than verified. The file counts the 134 as summary.nulls_or_worse against summary.confirms of 121, and its honest_ratio_nulls_per_confirm is 1.107 -- for every family currently confirmed, roughly 1.1 are null, not_testable or retracted. The four provisional families are not a final verdict either way yet, which is why they sit outside the 134 and inside the 138.

Five of the 259 families changed verdict as they were re-tested (summary.n_flipped=5). Table 2 lists each one with its run count and full verdict sequence; verdict_flip_anatomy.json's own confounds field cautions that five is a small, anecdotal set to be read as case examples, not as a flip rate.

three_in_four_fatigue (basketball_nba) moved from NULL_LOCAL at n=4762 (effect -0.916) to CONFIRMED_LOCAL at n=7222 (effect -1.213), both runs against the same corpus, player_boxscores_2024_25_2025_26.

reliever_3in3d_fatigue (mlb) took three runs against the same corpus, bullpen_relief_chains+savant_full__2023-2025, to settle: NULL_LOCAL at n=9054 (effect 0.01839, small and positive), NULL_LOCAL at n=8897 (effect -0.03538), then CONFIRMED_LOCAL at n=8927 (effect -0.04412). The recorded effect crosses from a small positive value to a small negative one between the first and second runs, before the third run confirms.

reliever_3in3d_fatigue__combined, a related but distinct family in the same sport, moved the other way: PROVISIONAL at n=26878 to NULL_LOCAL at the same n=26878, with no effect value recorded for either step, and its current status is null.

staff_dayafter_fatigue_chain (mlb) is the one flipped family with dated steps, and all three share the identical run_ts, 2026-07-10T08:23:40Z: NULL_LOCAL at n=2087 (effect 0.0789), then CONFIRMED_LOCAL at n=2160 (effect 0.3915), then back to NULL_LOCAL at n=2078 (effect 0.3266). Its current status is null; the middle confirmed step did not hold.

break_point_conversion_by_set_number (tennis) is the one flip that is not obviously about new data: both steps report the identical n (34145) and the identical effect (-0.017) against the identical corpus, slam_points_2011_2015, yet the first step's verdict is NOT_TESTABLE and the second is CONFIRMED_LOCAL. Nothing in the two recorded steps distinguishes what was measured; here the flip is a change in how the same measurement was adjudicated, not a change in the measurement itself.

Seven families carry a current status of retracted; verdict_flip_anatomy.json's retracted list names what ended each one (Table 3). Unlike the five flips, six of the seven retracted families have only a single history row -- they were not confirmed and later reversed, they were tested once and rejected outright.

Table 4 gives the row-level view: not families, but every recorded mechanism test, with mechanism_survival.json's survival_rate (n_confirmed divided by n_testable).

By category (mechanism_survival.json, by_category), official_referee mechanisms survive at the highest rate among groups with a double-digit testable count (9 of 12 testable, 0.75), and travel mechanisms are confirmed on all 4 of 4 testable rows. rest_fatigue survives least often among the larger categories (11 of 36 testable, 0.3056). weather_park has 1 testable row of 3 and it did not confirm -- too small a denominator to read as a rate.

Fwd Claim Scoreboard

The full claim-family scoreboard: 259 families, current status, and the flipped and retracted subsets.

5 Robustness and what would falsify this

verdict_flip_anatomy.json's own confounds field states plainly that five flips is 'a small, anecdotal set -- this is qualitative case study, not a rate,' and that a flip reflects more or re-cleaned data arriving, not that the earlier verdict was careless. This note treats the five flips the same way: as worked examples of what a verdict change looks like, not as an estimate of how often verdicts change across the wider ledger.

142 of the ledger's 287 history rows carry no run_ts. For those rows, this note -- and the source files themselves -- can report sequence, not elapsed time. staff_dayafter_fatigue_chain shows the risk directly: all three steps share one run_ts, so they establish no ordering by time beyond the sequence position already recorded.

The clearest falsification check available here is the definitional one described in Method: mechanism_survival.json and mechanism_ledger_export.json count confirmations differently on the identical 287 rows (129 versus 130) because of where the single label ARTIFACT_CONFIRMED sits. That a one-label reclassification moves the count is a reason to treat any single confirmed-percentage headline as a function of taxonomy choices, not a fixed property of the ledger.

A ledger that only ever moved toward confirmation would be a warning sign. This one does not. staff_dayafter_fatigue_chain reached CONFIRMED_LOCAL on its middle step and ended on NULL_LOCAL, a confirmation that was reversed. reliever_3in3d_fatigue__combined never recorded a confirmed step at all: verdict_flip_anatomy.json's flips[].steps[] holds exactly two entries for it, PROVISIONAL then NULL_LOCAL, so it moved from an unsettled verdict to a null rather than reversing a confirmation. Seven further families were retracted outright on FAILED_REPLICATION or REJECT verdicts. Movement runs in both directions in the same record, which is what a self-grading ledger should look like.

6 Limitations

• Five flips out of 259 families is too small a count to treat as a base rate for how often verdicts change; verdict_flip_anatomy.json's own confounds field calls this a qualitative case study, and this note treats it the same way.
• 142 of the ledger's 287 history rows have no run_ts, so 'first' and 'last' for those rows reflect source-file order, not a confirmed chronology; one flipped family's three steps share a single timestamp and establish no time ordering among themselves at all.
• The row-level 'confirmed' count is bucket-definition-dependent: mechanism_survival.json and mechanism_ledger_export.json count 129 and 130 confirmed rows respectively out of the same 287, a one-row difference driven entirely by whether ARTIFACT_CONFIRMED is placed in the confirmed bucket.
• A family's current 'verified' status is the validation ledger's own latest recorded verdict, not an outcome checked against a further, independent corpus beyond what that family's own history already records.
• The sport and category labels used here (including a 'basketball_nba' key and a separate 'nba' key in mechanism_survival.json, and the seven keyword-matched categories) are inherited from the source ledger's own grouping and are reported as recorded rather than reconciled or re-derived.

7 How to read this on the site

The verdict-flips finding walks through these same five families with the same sourcing, and the retraction finding covers the seven retracted families alongside the project's broader retraction record. The published figure for fwd_claim_scoreboard sits next to its full family list in the analytics library, and mechanism_survival's by-sport and by-category tables are worth reading in full.