The graveyard: what the null results teach

1 Question

CourtVision's candidate-signal pipeline rejects far more ideas than it ships. This note is about that rejection record itself: not which signal held up, but what kind of ledger keeps score when most proposals do not. Two separate, both-published ledgers are read here. honesty_exhibit.json tracks hypothesis tests inside four sport-specific validation ledgers plus a fifth, larger interaction-generation ledger, bucketing each outcome into confirmed, null, not_testable, failed_replication or other. reject_graveyard.json tracks a different pipeline stage: the signal-audit gate a candidate engineered feature passes through before a hypothesis about it is even framed, recording REJECT, SHIP, SHIP_REVIEW, DEFER and several rarer verdicts, plus a free-text reason a fixed set of regular expressions groups into a taxonomy. A third artifact, paper_execution_audit.json, checks briefly that the same discipline -- reporting a null rather than filling in a number -- holds outside the validation ledgers too.

This is not the companion piece to how-proposed-signals-survive-testing, which reads fwd_claim_scoreboard.json and verdict_flip_anatomy.json at the claim-family level: which hypotheses are currently verified, which flipped verdict on a re-test, and which were retracted outright. That ledger and this one track different objects -- tested hypotheses versus audited candidate features -- and this note does not re-report its five flips or seven retractions. The question here is narrower: across the reason categories a much larger reject ledger records, what does the shape of the failure taxonomy look like, and what does a search process that keeps and counts its own nulls avoid that one reporting only its wins cannot?

2 Data and definitions

honesty_exhibit.json is generated at 2026-07-22T20:26:10 UTC and reads four sport-specific validation ledgers (domains/basketball_nba, mlb, soccer and tennis, each a validation_ledger.jsonl of tested hypotheses) plus a fifth source, data/cache/intel_claims/interaction_factory_ledger.jsonl, labeled interaction_factory. Every tested row is bucketed into one of five outcomes, in the order the file defines: confirmed, null, not_testable, failed_replication, other. Its own headline states the pooled comparison across all five sources: nulls (351) outnumber confirms (168), a ratio it reports as 2.1x.

reject_graveyard.json reads a separate ledger -- scripts/platformkit/reject_ledger.py, backed by data/frontend/reject_ledger.jsonl -- that records every verdict logged for a candidate engineered signal, not a tested hypothesis. Its full recorded history holds 804 rows (full_history_row_count); collapsing each (sport, signal) pair to only its latest verdict leaves 68 distinct signals currently sitting on a REJECT-family verdict (latest_per_signal_graveyard_count). history_vs_latest_disclosure names the gap directly: signals retested more than once (as-of-reclaim sweeps, re-gating) plus rows whose latest verdict has since flipped to SHIP. Of the 804 rows, 628 carry a REJECT-family verdict (reject_row_count); the file's own note states that a REJECT or DEFER verdict is evidence the gate is doing its job, not a failure to hide.

paper_execution_audit.json checks that the same discipline holds one stage further along, in a paper-only execution record rather than a candidate-signal test. It holds 83 logged records (n_records), none of which were real fills (n_executed_filled is 0); 37 settled against a final score (status_split.settled), and for every one the file records realized_clv_pct as null, with a note (realized_clv_honest_note) that no independent closing-price feed was captured -- unmeasurable rather than backfilled.

3 Method

reject_graveyard.json's reason-category grouping is built by matching each rejected row's free-text reason field against an ordered list of regular expressions -- asof_reclaim_sweep, seed_stability, calibration_only_unproven, cross_corpus_adjudication, conformal_width, wf_ablation, prescreen_delta, sealed_holdout, planted_null and discovered_transform among them -- and a row matching none of them is labeled other, provided it carries any reason text at all. The 11 categories with at least one row in this snapshot (rejects_by_reason_category) sum to exactly 628, the full reject_row_count, so the taxonomy is a complete partition of the reject-family rows rather than a sample of them.

Alongside the reason taxonomy, reject_graveyard.json separately reports verdicts_by_type across the full 804-row history: REJECT (627) and DEFER (1) together make the 628 reject-family total; the remainder are SHIP (95), SHIP_REVIEW (45, a conditional pass flagged for review), NOT_TESTABLE (14), WORSE (6, worse than the standing baseline), INSUFFICIENT (6), NOT-REPLICATED (4), STANDING_BASELINE_SET (3, a baseline-registration row rather than a candidate verdict), TIER3_BLOCKED (1), UNDERPOWERED (1) and NULL_UNDERPOWERED (1).

honesty_exhibit's five outcome buckets and reject_graveyard's twelve verdict types are not the same taxonomy applied to the same rows; they describe two different pipeline stages, kept in two different ledgers, and this note does not add their counts together. A hypothesis reading null in the validation ledger and a feature reading REJECT in the signal-audit ledger are both, in the ordinary sense, things that did not survive testing -- the common thread this note follows, not a claim that the two counts describe one combined population.

4 Results

Table 1 reads reject_graveyard.json's own reason-category counts, the finest published grain for why a candidate signal was rejected.

asof_reclaim_sweep alone accounts for 510 of the 628 reject-family rows, 81.2% of the total: not 510 independently discovered dead ends, but what a bulk re-validation sweep produces when it re-checks previously recorded rows against the as-of rule and reclassifies them. The next four categories -- prescreen_delta (20), wf_ablation (18), seed_stability (17) and other (14) -- read closer to one-off checks, nearer to a single candidate signal's specific failure. The remaining six categories (calibration_only_unproven, discovered_transform, sealed_holdout, conformal_width, cross_corpus_adjudication, planted_null) are each under 15 rows, none reaching 3% of the total on its own.

Underneath the two largest verdict types (REJECT at 627 and SHIP at 95), the ledger records rarer outcomes that are not simple pass or fail. Table 2 lists the four verdict types appearing exactly once in the 804-row history; reject_graveyard.json does not publish a per-signal id, sport, corpus or date broken out by verdict type, so the table states that directly rather than estimating a split the artifact does not carry.

Table 3 brings honesty_exhibit.json's own per-ledger counts into view in full, the basis for the pooled 351-versus-168 headline.

The argument for publishing this table rather than only the confirmed rows is structural, not a claim about any particular signal: a process that reports its confirms and discards its nulls will always look stronger than one reporting both, at the same true rate of success. honesty_exhibit.json's pooled count -- 351 null against 168 confirmed -- is the concrete form: if those 351 rows were simply not recorded, the ledger would read as a strong track record rather than the ordinary base rate a search over hundreds of hypotheses produces. The reject ledger makes the same point at a larger scale: 628 of 804 recorded verdicts are REJECT-family, and even after collapsing reruns to each signal's latest test, 68 signals are still sitting on a rejected verdict -- what makes the 95 SHIP and 45 SHIP_REVIEW verdicts elsewhere in the same ledger legible as a rate rather than a chosen-after-the-fact count.

The interaction_factory source makes the scale argument sharpest: of its 1003 rows, 709 (70.7%) are not_testable and only 38 confirmed. Reporting that ratio, rather than surfacing only the 38, treats the 709 not-testable rows as the recorded cost of searching this widely, not evidence of nothing.

paper_execution_audit.json shows the same discipline one stage further downstream, where a claim has already been logged for placement rather than tested as a hypothesis: of 83 paper-only records, 37 settled by final score, and realized_clv_pct is recorded as null for all 37 rather than estimated, because -- in the file's own words -- no independent closing-price feed was captured. A missing number reported as missing is the same habit as a null reported as null.

Reject Graveyard

The reject ledger's reason-category counts (628 rows) beside its cumulative-rejects-over-time curve (804-row full history, 2026-06-09 through 2026-07-18).

5 Robustness and what would falsify this

The reason taxonomy is regular expressions matched against a free-text field, documented as good enough for a readable breakdown rather than a hand-labeled ground truth, with a dedicated taxonomy field named as the upgrade path. A different regex ordering, or a hand audit of the 14 other-labeled rows, could move rows between categories without changing the 628 total.

reject_graveyard.json does not publish a sport or date for any individual reason category or verdict type; only whole-ledger totals (history_rows_by_sport, cumulative_rejects_over_time) are broken out that way. This note does not construct a per-category split by combining fields the artifact keeps separate, and Table 2 says so directly rather than estimating one.

The 68 distinct signals in latest_per_signal_graveyard_count is a snapshot of the latest verdict per (sport, signal) pair, not a permanent count: history_vs_latest_disclosure states that part of the 804-versus-68 gap is rows whose latest verdict has since flipped to SHIP, so a signal counted among the 68 today can leave the graveyard on a later retest, the same way claim families flipped verdict in the companion paper's reading.

honesty_exhibit's five-way split and reject_graveyard's twelve-way split are read side by side because both describe things that did not survive testing, but they are not the same measurement of the same population, and this note does not sum a null hypothesis and a REJECT signal into one combined count.

6 Limitations

• The reason-category taxonomy in reject_graveyard.json is assigned by regular expressions over a free-text reason field, not a hand-audited label; the source script calls it good enough for a readable breakdown, naming a hand-labeled taxonomy field as the upgrade path.
• asof_reclaim_sweep alone is 81.2% of the 628 reject-family rows and is a bulk re-validation sweep; a taxonomy this dominated by one housekeeping category is not a like-for-like picture of 628 independently discovered failure modes.
• reject_graveyard.json publishes sport and date only as whole-ledger totals, not crossed with reason category or verdict type; Table 2 reports that limit directly rather than estimating a split.
• latest_per_signal_graveyard_count (68) is a snapshot as of this file's generation; it falls as retested signals ship and rises as new ones are gated, so it is not a running lifetime count of failed signals.
• honesty_exhibit.json and reject_graveyard.json are two different ledgers over two different objects, tested hypotheses versus audited candidate features; their bucket counts sit side by side for the shared argument about publishing nulls, not added into one combined rate.

7 How to read this on the site

The reject_graveyard module page carries the full reason-category and cumulative-rejects figure referenced above; honesty_exhibit's page holds the per-ledger breakdown behind Table 3 in full, including the interaction_factory source. The companion paper, how-proposed-signals-survive-testing, covers the claim-family-level verdict flips and retractions in the validation ledger that this note deliberately does not re-report; how-to-read-a-courtvision-paper explains the evidence-citation convention both papers follow.