Skip to content
Browse › Qa Coverage Stats
Analytics module · as of 2026-07-22

QA coverage: fail-closed answer engine, all checks green

answer engine: 87/87 regression-bank checks pass (fail-closed statuses graded as PASS); honest coverage 36.6% of answerable questions -- refusals (no_data/not_supported/ambiguous/refused) are the fail-closed design, not a defect
View full size ↗
confirmednull (a finding)not testabledescriptivepending
Qa Coverage Stats
Chart: Qa Coverage Stats -- answer engine: 87/87 regression-bank checks pass (fail-closed statuses graded as PASS); honest coverage 36.6% of answerable questions -- refusals (no_data/not_supported/ambiguous/refused) are the fail-closed design, not a defect
scripts/platformkit/analytics_showcase/out/qa_coverage_stats.json2026-07-22

What it means

Every one of 87 regression-bank checks passes (fail-closed statuses count as PASS), and on a 1,307-row coverage stress test the engine answers 36.62% of the questions it deems answerable and declines the rest. Every one of 125 edge-language questions was refused (ok_rate 0.0), which is exactly the intended behavior.

Caveats & confounds

The 36.6% figure counts only rows flagged expects_answer=true; a large share of real questions still resolve to no_data (560 of the 1,307 stress rows).

Ask Scout about this
Which sport is best covered by the answer layer?How do you test the answer layer itself?How many questions can the system actually answer?Ask anything →