We ran RuleReceipt against a real, long, complex session — not a demo, not a curated example. It found real bugs, in our own tool, before anyone else did. Here's exactly what was wrong.
Every single FAIL that session produced was wrong. Not "arguably wrong" — verifiably wrong, checked by hand against what actually happened. So we stopped, found the actual causes, fixed them, and are publishing exactly what happened instead of quietly patching it.
A rule like "never leave a print( statement in committed code" was checked by searching the entire session transcript for the literal text print( — including the output of commands, not just the agent's own edits.
print( in a file it never touched. Reported as a violation.A rule requiring tests for code changes was checked by looking for ANY edited file that wasn't itself a test file — including markdown docs, JSON, and YAML.
.md, .json, .yaml, and similar) are excluded from this check entirely.Our own global rule, "tests must be able to fail" — the exact rule behind every red-run proof in this project — was misclassified as "every edit needs a test file," because the classifier just looked for the word "test" plus a generic word like "must" anywhere in the rule.
Fixing three bugs wasn't enough, because chasing individual false positives is itself the mistake. So we built a test harness and ran the classifier over 559 real public rule files — CLAUDE.md, AGENTS.md, AGENT.md, .cursorrules, Copilot instructions, .windsurfrules and GEMINI.md — from projects like PyTorch, Kubernetes, and Elasticsearch. 21,986 parsed items, of which 8,043 were real rules. Two things came out of it.
Only about a third of what the parser extracted contained any instruction at all. The rest was documentation: directory listings, model-reference tables, import examples, glob-syntax notes. And we were running checks against all of it. Asking whether a session "violated" a directory listing has no meaningful answer — every coincidental word match became a reported finding.
Our first attempt at fixing this was to detect what documentation looks like. That was wrong too, and the corpus proved it: coverage fell sharply when we moved from a 40-file sample to a corpus more than ten times the size, purely because bigger samples use formatting we hadn't listed yet. Enumerating documentation shapes is an infinite list. So we inverted it — test for whether a line contains an instruction, which is a bounded question about language and doesn't change when someone invents a new markdown convention.
Searching a transcript for a banned string tells you the string appeared. It cannot tell you the agent did the forbidden thing — the identical match is produced by grepping for that pattern, quoting it in an explanation, or naming it in a commit message. The information needed to tell an action from a mention simply isn't in the text.
An independent test against a real session confirmed it precisely: 19 of 19 remaining failures were repo and branch names in ordinary conversation or inside legitimate git commands. Zero were real violations.
So we changed the rule permanently, for every check, in every format: a text match never produces a failure. It reports what it found, quotes it, and says plainly that it can't distinguish a violation from a mention. Confident failures now come only from structured checks that read what was actually executed or written — an actual git command's actual branch argument, actual file edits, actual file operations. Four of those exist so far; more will follow, one category at a time.
The measurable result: the fragile keyword path — the source of every false positive on this page — dropped from 43.5% of real rules to 29.6%, and to 10.8% of everything the parser extracts, and the remainder can no longer report a false failure by construction. Verified across all 559 files, with no crashes. The corpus was later grown roughly fourteen-fold, from a 40-file sample to 559 files across formats the classifier had never been tuned on, and the distribution barely moved: lines carrying no instruction went from 65.4% to 63.4%, and the share of real rules answerable mechanically went from 47.8% to 44.2%. That stability is the actual argument: directives are a property of language, not of file format. The checkable figure has since fallen to 35.3% for a different reason — see the last row below — because part of what it was counting turned out to be the classifier guessing.
Every figure on this page is a measurement taken on a date, with a classifier that keeps changing. Overwriting them each time would hide the thing most worth seeing — which corrections moved which number, and by how much. So the previous values stay.
| Measured | Files | Parsed | Not a rule | Real rules | Checkable | What changed |
|---|---|---|---|---|---|---|
| 30 Aug 2026 | 40 | — | 65.4% | — | 47.8% | First sample, after the false-positive rewrite above. Absolute counts weren't recorded at the time. |
| 31 Aug 2026 | 559 | 23,704 | 62.9% | 8,803 | 43.5% | Corpus grown roughly fourteen-fold, into formats the classifier had never been tuned on. The distribution barely moved, which is the argument. |
| 4 Sep 2026 | 559 | 23,704 | 64.5% | 8,413 | 42.7% | Command documentation was being counted as rules. A line documenting a flag describes what the flag does, and uses the same verbs as an instruction, so the directive test could not separate them. 390 items reclassified. |
| 4 Sep 2026 | 559 | 21,986 | 63.4% | 8,043 | 44.2% | The parser had no concept of a fenced code block, so a shell comment starting with #, or a YAML item starting with -, was read as a rule boundary. 588 rules were being cut mid-block, and 1,718 items — 7.2% of everything extracted — were lines lifted out of code samples that had never been rules at all. Unclosed fences now 8. |
| 12 Sep 2026 | 559 | 21,986 | 63.2% | 8,101 | 35.3% | The largest correction so far, and it moved the checkable figure down. When the classifier could not tell whether a rule forbade something or required it, it defaulted to forbid — the accusing direction — on the grounds that this preserved existing behaviour. It failed a real session for following Cypress’s own rule to “update the closest relevant CLAUDE.md”, because a CLAUDE.md had been updated. Direction is now never guessed: undetermined rules go to human judgment. Those 8.9 points were rules the tool was guessing about, so they were never really checkable — the earlier figures counted guesses as coverage. Also fixed: literals that cannot identify anything (, appeared 22 times as a checkable pattern and matched every file containing a comma), and a missing leading word boundary that fired rm inside “performance”. |
| 14 Sep 2026 | 559 | 21,986 | 63.2% | 8,101 | 34.8% | A smaller correction, listed because the figure moved. Claim-evidence checks were being routed whole document sections — 51 of the 83 rules reaching that path had bodies over 1,500 characters, against a median of 71 for rules generally, and one titled “AutoEvolve Instructions for GitHub Copilot” produced 166 false failures on its own. Sections that long are no longer treated as a single rule. Same release closed a one-line fault worth 140 more: content matching chose its route if any backtick literal looked like code, then searched for every literal, so a rule mentioning console.log( also matched any file containing the word name. Across 559 rules files run against 5 real sessions, the share of reports carrying a false accusation fell from 15.8% to 2.9%. |
| 15 Sep 2026 | 559 | 21,986 | 63.2% | 8,101 | 35.2% | Four routing faults, all found by moving the same checks from a report into a blocking gate. Nobody notices a wrong line in a report; a wrong line in a gate stops real work, so each one had to be chased down. Branch policy took the first backtick literal in any rule mentioning “branch” and treated it as a branch name — 37 of the 70 rules routed there (52.9%) were handed something that cannot be one, including a naming template and an entire fenced markdown block; now 0 of 45. Claim-evidence required a reporting verb, an evidence noun and a done-word to appear independently anywhere in a rule, which a 944-character rule about writing status updates satisfied across three separate paragraphs; it now needs the verb beside the done-word in one sentence, narrowing that path from 38 rules to 8. Content matching treated any literal containing a parenthesis as code, so (in the was searched for in written files — 74 of 1,086 such literals (6.8%) were prose, now 0. And it matched by bare substring, so a rule forbidding fetch() fired on _metar_fetch(). The coverage figure moved 0.4 points on these reroutes; the cause was not isolated further, because rules moved between checkers rather than in or out of coverage. Separately: the harness behind the previously published 15.8% and 2.9% was not kept, so those two cannot be reproduced and should be read as history. Its replacement pins and prints its inputs, and reports 30 of 2,795 reports carrying a false accusation, 1.1%. |
The last row also came with a finding that does not fit in a table. The classifier looks for a list of English instruction words, and this project's own rule — “No silent scope changes” — had never been checked by it, because the rule says “say so explicitly” and say is not on the list. It had been dropped as documentation in every report this tool had produced.
That gap does not close by lengthening the list. Imperative verbs are
not a closed class, and a list of English ones cannot match a rule written
in another language — measured across the same 559 files,
non-English content is dropped at 97.5% against a 64.5% baseline. What
closes is the silence: rulereceipt check --show-skipped now
lists what was excluded, so a rule the classifier failed to recognise is
something you can see rather than something you never hear about.
A tool whose entire purpose is "don't just trust the summary, show the evidence" doesn't get to ask for trust it isn't willing to extend to itself. We found these by actually using the tool on real work, the same way we're asking you to. Now you know exactly what was wrong, exactly how it was found, and exactly what changed.