The worst number we ever measured

A yardstick before a fix

The first thing we built was not a feature. It was a benchmark: a frozen corpus of official government pages, gold facts hand-written against them and quoted verbatim from the source, and a sealed holdout the system never sees while it is tuned. A separate grader — never the model being graded — scores every run. Floors instead of averages: a run below the recall floor fails even when its average looks respectable, because in tax the fact you miss is the one that matters.

The benchmark found what no demonstration would have found. A page cleaner was keeping script bodies, so headline rate tables survived only as buried metadata; one fix lifted table-fact extraction sharply and brought anchoring to a perfect score. Evidence windows were too narrow to carry a table header, so our strictest verifier kept refusing true facts — the defect was ours, not the model’s. Headline rates live in tables and lists, and early on the system threw them away rather than risk a malformed read. We rebuilt the reader for tables and lists, blind-judged the output, and only then switched it on.

Seven of twenty-nine

Then we ran the whole pipeline end to end on a real jurisdiction and let the independent judge score what came out. Twenty-nine facts promoted. Seven judged ready to serve.

The gate refused to ship it, and we let it. The run report named nine defects, with file and line numbers, instead of scaling the failure. We shipped nothing that month. We kept the number.

The road, in five moves

• Refuse more than you accept. A family of deterministic refusal rules now guards promotion: a quote must be a whole sentence, a period must sit inside the clause that carries the value, worked examples are excluded, statistics pages are not law. What the system cannot support, it flags for a human reviewer — it never guesses.
• Re-measure for free. Every paid run is recorded as a trace, and every fix is re-gated against those traces at zero marginal cost. Iteration stopped being expensive, so it stopped being rare.
• Judge blindly, judge separately. The model that answers is never the model that scores, and nothing switches on by default until a blind judge clears it. The judge’s own verdicts are sampled and reviewed.
• Read what people actually read. One of our test jurisdictions writes its tax law in a right-to-left script, and a rule fitted to left-to-right text silently refused 116 true facts across eight pages. The fix took those refusals to zero on the same bytes — proven with a replay that cost about twelve cents.
• Keep the worst numbers published. When a run fails its gate, the failure is the finding. The low scores are as much a part of the record as the high ones, and they are dated the same way.

Where it stands

The same archived pages, judged the same blind way, against naive AI models: from 24% to 98%.

Caveats are part of the claim. These figures are measured on a fixed benchmark of official pages, under a grader that never saw the answering system’s output. We publish the floors, not just the highs.

Seven of twenty-nine was the day we started telling the truth about ourselves. Everything since is what telling the truth bought.

Why the model was never the product

Every validated rate or threshold fact requires Tier-1 evidence: tax authority, legislation, or treaty text. That sentence is a rule the system enforces, not a promise we make. The frontier model was roughly the same through this whole story; what changed is everything around it — the corpus, the gold, the refusals, the gates, the judges. That machinery is the product, and it is why a getax report can be defended line by line, with every number anchored to official text a reviewer can open and read.

You could ask a frontier model and receive a plausible sentence. The sample report shows what an answer looks like when the machinery around the model has done its work.