We ran RuleReceipt against a real, long, complex session — not a demo, not a curated example. It found real bugs, in our own tool, before anyone else did. Here's exactly what was wrong.
Every single FAIL that session produced was wrong. Not "arguably wrong" — verifiably wrong, checked by hand against what actually happened. That's the kind of result that should stop a launch, not follow one. So we stopped, found the actual causes, fixed them, and are publishing exactly what happened instead of quietly patching it.
A rule like "never leave a print( statement in committed code" was checked by searching the entire session transcript for the literal text print( — including the output of commands, not just the agent's own edits.
print( in a file it never touched. Reported as a violation.A rule requiring tests for code changes was checked by looking for ANY edited file that wasn't itself a test file — including markdown docs, JSON, and YAML.
.md, .json, .yaml, and similar) are excluded from this check entirely.Our own global rule, "tests must be able to fail" — the exact rule behind every red-run proof in this project — was misclassified as "every edit needs a test file," because the classifier just looked for the word "test" plus a generic word like "must" anywhere in the rule.
Fixing three bugs wasn't enough, because chasing individual false positives is itself the mistake. So we built a test harness and ran the classifier over 948 real public rule files — CLAUDE.md, AGENTS.md, AGENT.md, .cursorrules, Copilot instructions, GEMINI.md — from projects like React, PyTorch, Kubernetes, and Elasticsearch. 35,755 rules. Two things came out of it.
Only about a fifth of what the parser extracted contained any instruction at all. The rest was documentation: directory listings, model-reference tables, import examples, glob-syntax notes. And we were running checks against all of it. Asking whether a session "violated" a directory listing has no meaningful answer — every coincidental word match became a reported finding.
Our first attempt at fixing this was to detect what documentation looks like. That was wrong too, and the corpus proved it: coverage fell from 17% to 7% when we moved from a 40-file sample to a 658-file one, purely because bigger samples use formatting we hadn't listed yet. Enumerating documentation shapes is an infinite list. So we inverted it — test for whether a line contains an instruction, which is a bounded question about language and doesn't change when someone invents a new markdown convention.
This is the one that mattered most. Searching a transcript for a banned string tells you the string appeared. It cannot tell you the agent did the forbidden thing — the identical match is produced by grepping for that pattern, quoting it in an explanation, or naming it in a commit message. The information needed to tell an action from a mention simply isn't in the text.
An independent test against a real session confirmed it precisely: 19 of 19 remaining failures were repo and branch names in ordinary conversation or inside legitimate git commands. Zero were real violations.
So we changed the rule permanently, for every check, in every format: a text match never produces a failure. It reports what it found, quotes it, and says plainly that it can't distinguish a violation from a mention. Confident failures now come only from structured checks that read what was actually executed or written — an actual git command's actual branch argument, actual file edits, actual file operations. Three of those exist so far; more will follow, one category at a time.
The measurable result: the fragile keyword path — the source of every false positive on this page — dropped from 43.5% of real-world rules to about 10%, and the remainder can no longer report a false failure by construction. Verified across all 948 files, with no crashes.
A tool whose entire purpose is "don't just trust the summary, show the evidence" doesn't get to ask for trust it isn't willing to extend to itself. We found these by actually using the tool on real work, the same way we're asking you to. If we'd hidden this, the next person to hit it would have had to find it themselves, with no idea whether it was a one-off or the whole thing being unreliable. Now you know exactly what was wrong, exactly how it was found, and exactly what changed.