General
Verifying every number in a draft against its source does not make the draft honest.
Michael Lip Dev.to (EN Zone)
1 views
Verifying every number in a draft against its source does not make the draft honest.
I learned that on my own writing pipeline. The gate I run reads each draft, pulls out every figure, and traces it back to the measurement that produced it. If a figure has no source, the draft doesn't ship. That part works. The gate has never let a made up figure through.
Then I read eight drafts that had passed clean, and every one of them claimed work I never did.
One of them opened like this.
Three weeks ago, I pointed a scraper at TikTok and
I never pointed a scraper at anything. There's no scraper. But the gate had nothing to object to, because that sentence carries no figure at all. It's a claim about me, not about a quantity, and the gate only knows how to check quantities.
The harder ones were the drafts where both figures were real. A draft would cite two numbers that traced back cleanly to the measurement behind them, then wrap them in a story about an experiment that never happened. Every digit held up. The verb didn't. The gate scored the draft as fully sourced and passed it.
What I counted
I went back through the run and read every draft by hand. 8 of 79 claimed work I never did. All of them read as confident and specific. All of them were clean by the only measure I had.
The chart at the top of this page shows that split.
That reread is what it cost me. Seventy nine drafts, one at a time, against my own record of what I actually ran. Eight of them went in the bin. And the number check I'd trusted for months turned out to be covering a narrow slice of the thing I thought it covered.
What I changed
The gate still checks every figure the way it always did. It now also pulls out every first person action verb in the draft. Ran, built, deployed, scraped, measured, pointed.
Each of those actions has to match an event in my record of what I ran. If the record holds nothing that matches, the draft fails.
That's a much harder check to write. A figure is a token you can match. An action is a sentence you have to understand, and my record doesn't use the draft's words. So I get false refusals now. A draft says I measured something, the record calls it a read, and the gate stops a true sentence.
I'll take that trade. I'd rather refuse an honest draft than ship one more sentence about a scraper I never ran.
The lesson is narrow, and I think it's worth having anyway. A check covers what you pointed it at, and nothing next to it. Mine was pointed at the figures. So the figures were the one part of those eight drafts that was true.
Read original: https://dev.to/alphashark/verifying-every-number-in-a-draft-against-its-source-does-not-make-the-draft-honest-3m55
← Previous
Claude 3.5 Sonnet is more than an upgrade. It’s a new workflow.
Next →
Postgres RLS in Symfony: three green isolation tests, and the UPDATE that moves tenant 1's invoices into tenant 2's books
Related
What "Fully Automated" Actually Costs
General
0
DEV Community
Ethernet Speed Evolution Reaches 1.6 Terabit Milestone
General
0
DEV Community 周榜
I wrote three rules on Saturday. I broke all three on Saturday.
General
0
DEV Community 周榜
How I Built Calculadora SIU CrediUPE: Automating Credit Calculation in SIU Guaraní
General
1
DEV Community 周榜
Comments0
No comments yet — be the first