"Does your AI code reviewer read CLAUDE.md?" is the wrong question to ask. Almost any of them will say yes, and honestly, most will catch something. What actually matters is whether you can tell the difference between a tool that read your rule and a tool that just happens to agree with it. So I built a small API, wrote real rules into CLAUDE.md and AGENTS.md, and then broke one of those rules twice on purpose. The first break was something any decent security scanner would flag anyway. The second one had no reason to get caught unless a tool actually opened the file and read it. I ran both through CodeRabbit and Qodo to see what would happen. Both tools caught the second one eventually. Only one of them could tell me why. the setup rules-demo-api is a small Cloudflare Worker. POST /feedback takes a rating and a message, validates them, and logs the rating. Both CLAUDE.md and AGENTS.md document the same four rules for the repo, including this one: "Never log request headers or bodies. They may contain secrets, tokens, or PII." and one I added later: "All new route paths must be kebab-case (/feedback-summary), never camelCase (/feedbackSummary). This is an internal convention, not a general best practice." I wrote that second rule that way on purpose. There's nothing about kebab-case routes that's a security issue, and nothing about it is a bug either — it's just a house style choice with zero backing outside this one repo. Which is exactly what makes it a cleaner test than the first rule. If a tool flags a violation of this one, it had to have read the file, because there's nothing else it could be catching. It's a small example, but this is exactly the kind of gap that shows up once you're running AI code review at scale instead of just eyeballing one repo yourself. test one: the rule with a built-in excuse I opened a PR that logs the full request headers and body, with a commit message saying it was "temporary, to debug a client issue." Both tools caught it right away. Here's what Qodo said: Rule violation, High: Customer secrets enter application logs "Rule 1: Do Not Log Unredacted Request Headers or Bodies" (AGENTS.md[5-8]) And CodeRabbit: "This change can expose credentials and private feedback content through application logs. It should not merge until logging is restricted to an allowlist of necessary diagnostic fields." Merge Risk: High. Both correct, and only one of them names the actual file. But this test doesn't really prove anything about file-reading either way: logging raw headers and bodies is a well-known bad practice whether or not you've written it down anywhere. Any half-decent scanner flags this on instinct alone. Call this round a wash. test two: the rule with no excuse Same repo, new PR. This time I added a GET /feedbackSummary endpoint in camelCase, breaking the kebab-case rule and nothing else. No security issue, no bug, just a naming convention nobody outside this repo has ever heard of. The only way to catch it is to have actually read CLAUDE.md or AGENTS.md. Qodo caught it immediately: Rule violation, High: Clients cannot use the kebab-case route "Rule 4: Use Kebab-Case for New API Route Paths" (AGENTS.md[7-10], CLAUDE.md[7-10]) "Requests to /feedback-summary... do not match the condition and fall through to the handler's 404 response." It even took the reasoning a step further than I expected. Since the handler 404s on anything unmatched, Qodo worked out that a client calling the correct kebab-case URL would get a 404, while my wrong, camelCase URL worked fine. That's not just matching my code against a style guide. That's tracing what the naming mismatch does once real requests hit it. CodeRabbit, running on its default "Chill" profile, had a different take: "No actionable comments were generated in the recent review. 🎉" Merge Risk: Minimal. Nothing. A clean bill of health on a PR that breaks a rule written into both files in the same repo CodeRabbit had just finished reviewing. the profile problem Before I wrote CodeRabbit off, I went and checked its settings. There's a Chill/Assertive toggle in there, and Assertive is documented to surface more style-level feedback. So I switched it over and forced a full re-review on the same PR. This time, it caught the naming issue: "Update the route condition... to match the required kebab-case path /feedback-summary instead of /feedbackSummary." No mention of CLAUDE.md. No mention of AGENTS.md. No rule number, nothing. Just "the required kebab-case path," stated like a fact everyone already knows. And that's fair, actually, because kebab-case for REST routes really is a common convention across the web, even though I'd framed it here as an arbitrary house rule specific to this repo. So CodeRabbit catching it on Assertive doesn't prove it read my file at all. It's just as easily explained by a stricter style-sensitivity dial reaching for a convention it already knew before it ever saw my repo. I picked a rule I thought had zero generic backing, and I got that part slightly wrong. Better to admit that than pretend the test was cleaner than it was. what separates them It's not catch versus miss. Both tools eventually caught the same thing. What's different between them is whether that catch is checkable. https://youtu.be/_QlBOdiATxU Every single Qodo finding across both PRs, no matter the severity, points to a specific rule number and a line range in a named file. Each finding also carries its own severity rating, the exact code it's flagging, the reasoning behind the call, and a fix prompt you can hand straight to your own agent. You can click the link yourself. You can go read the rule and confirm it says what Qodo claims it says. I don't have to take any of that on faith. CodeRabbit's findings are real, and often genuinely well-worded, but they never show their source. Whether a given catch came from actually reading AGENTS.md, or from a stricter profile dial turned up one notch, there's just no way to tell from what it hands you. Flipping Chill to Assertive proves the sensitivity is configurable. It doesn't prove the source of any single catch is your file. At one repo, honestly, that difference barely matters. You'll eyeball the PR either way. But once you're running this across dozens of repos, with rules that get written and updated by different people over time, "did the reviewer actually check the current rule" stops being a question you can shrug off. An unauditable catch and an unauditable miss look exactly the same from the outside. You won't know which one you're getting until it's already gone wrong. takeaway Ask a different question than "does it read CLAUDE.md." Ask "can it show me the line it read." One of these tools answers that on every single finding, no exceptions. The other never answers it, no matter how many settings you turn up. That's the dimension actually worth comparing tools on, more than any feature list — see AI code review tools comparison and benchmarks if you want the fuller picture beyond just these two. A rules file sitting in a repo is best-effort compliance at best: readable by a human if they bother to look, but enforced inconsistently depending on whatever's reviewing it that day and whatever profile happens to be set. A rule that's centrally managed and cited by line number is a different thing entirely. Qodo actually imports rules straight out of AGENTS.md and CLAUDE.md into its own centrally managed Review Standards, where each one picks up a severity and a scope of its own. It's auditable across every repo it touches, whether that's one repo or two hundred of them. Qodo's own Agentic Toolbox, launching September 9th, pushes this a step further. One of its skills, qodo-get-rules, loads the workspace's centrally-managed rules straight into the agent's session before it writes a single line, not just at review time after the fact. If a rule is worth citing by line number in a PR, it's worth putting in front of the agent before the PR exists. I hid a rule in a file that most tools claim to read. Getting caught wasn't the hard part. Proving it wasn't a coincidence was. Repo: rules-demo-api. Both tools run on their respective free/trial tiers for public repos.