What the Ask AI Can and Cannot Tell You

The Ask AI is not a chatbot improvising over some documents. It is a relay over a fail-closed engine. Every question routes to exactly one deterministic resolver chosen by rule, and the answer comes back as a typed envelope the model must honor verbatim: a status, the value, a sample size, a p-value, the exact source file, an as-of date, and edge_claimed=false. There is no room to round a number up, borrow a stale one, or invent a dollar edge. That structure is what lets it do three useful things and refuse three others honestly.

What it can do, tier one, is resolve a fact from a committed artifact — a verified analytics claim, a calibration number, a descriptive entity card — deterministically, the same number every time, because the same executable contract returns byte-identical results with no model call at all.

Tier two, it can return a mechanism receipt from the validation ledger. Ask whether the back-to-back rest penalty holds up in the NBA and it answers CONFIRMED_LOCAL with an effect of -1.73 points of margin, n=4,732, p about 0.0056, and the ledger file the finding lives in — and the envelope's own framing field caps it at a local single-corpus finding, not a market-beating or causal claim. It confirms folklore and refuses to inflate it in the same breath.

Tier three, and the most telling, it can reprice a game in-game and tell on itself. Ask for a live NBA win probability at a given score and it returns the repriced number and, in the same envelope, the calibration for that game-state bucket: over 615 games, model Brier 0.2328 against market Brier 0.1985. The market is sharper in that spot, and the engine says so, out loud, unprompted. A system that reports its own losses against the benchmark is one you can trust when it reports a match.

What it cannot do is answer from thin air, and it refuses in three distinct ways rather than guessing. It returns no_data when the backing artifact is absent — there is no live feed here, so tonight's odds, tonight's score, tomorrow's lineup, and any sport outside NBA, MLB, soccer, and tennis all come back as an honest no_data, not a borrowed number. It returns not_supported when no resolver is registered for the question type; the engine does not invent a handler. And it returns refused when the question is phrased in edge, ROI, pick, or profit language — in the coverage stress test all 125 edge-language questions were refused, an ok-rate of exactly zero, enforced by the no-edge rail rather than left to the model's discretion.

The honest part is that the gaps are measured and published, not hidden. Across a 1,307-question stress bank the status mix was 399 ok, 560 no_data, 42 not_supported, 181 ambiguous, and 125 refused, with zero errors, for an answerable-question coverage rate of 0.3662. Separately, a fail-closed regression bank runs 87 checks and passes all 87, where a correct no_data or not_supported counts as a pass. The design goal was never to answer everything. It was that a connected model cannot fabricate a number, and the honest gaps are visible and counted. That is the difference between an engine you can quote and a chat window that hopes.

Sources