Cloud
My negative test stopped being negative when a config file grew by four rows
Mahiro Hirakawa Dev.to (EN Zone)
2 views
A test whose entire job was to prove a rule gets refused had quietly started proving that the rule passes.
It plants a violation and asserts the checker rejects it. It had been green for weeks. It was green because the assertion had inverted, and nothing about a green test tells you which side of the assertion it is standing on.
expected: refusal (the planted row breaks monotonicity)
actual: OK_K3_MONOTONE steps=11 regress=0
result: the test now fails, correctly, for the first time in weeks
What happened
The test planted its contradiction at a fixed position, chosen because that position was the last row of a config table when the test was written.
The table had seven rows then. Another change grew it to ten. A later one grew it to eleven.
The planted violation was no longer last. The rows that came after it re-established the property the violation was supposed to break. The checker looked at the whole sequence and correctly said it was fine. The test asserts a refusal, and no refusal arrived.
Nothing was broken by that. The checker was right. The config was right. The test was measuring a scenario that had stopped existing.
Why nobody noticed
The changes that grew the table ran their own module's tests and the shared control. They did not run this one, because there was no visible relationship between "add rows to a config table" and "a test in another module about ordering".
That relationship existed, and it was invisible precisely because it was a literal:
// what it did
let planted = rows[7]; // "the last row", in the world where there were 8
// what it does now
let planted = rows.last(); // read from the config, whatever length it is
A literal copied out of a config file is a dependency with no edge in any graph. The build does not know about it. The type system does not know about it. A grep for the config's name does not find it, because the file is never mentioned. Only one number from it is.
The general shape
A negative test has two failure modes and they are not symmetric.
failure
what you see
how long it survives
it stops passing
red
until someone fixes it, usually within a day
it stops being negative
green
until someone reads it
The second one is worse in every way that matters. It takes the exact form of success, it is produced by a test that runs and exercises real code, and coverage tooling counts it as covered. The only thing it no longer does is fail when the thing it guards regresses.
Two repairs, one of which generalises
Derive the position instead of writing it. The planted violation now goes after the last declared row, read from the config at test time. The scenario cannot drift out from under the assertion because the assertion no longer names a coordinate.
Census the test files for literal fragments of config. That is the part worth stealing. If a test embeds a value that lives authoritatively somewhere else, that is a silent coupling, and a sweep for those is cheap:
$ # every numeric literal in tests that also appears as a row count in config
$ ... | wc -l
3
Three, in that codebase, and this was one of them. The other two are latent: correct today, in the same way this one was correct for weeks.
What I would check in your negative tests
Does the failure it plants get its position, its size, or its identity from a literal? If yes, something else owns that value and can change it without ever touching the test.
And: has this test ever failed? A negative test that has never once gone red has never demonstrated that it can. Flip its assertion deliberately, watch it fail, flip it back. That takes a minute and is the only evidence that the green means anything.
Read original: https://dev.to/mahirhir/my-negative-test-stopped-being-negative-when-a-config-file-grew-by-four-rows-1hi0
← Previous
Snapshot Exit, Stdout, and Stderr Before One Flag Extract
Next →
sync = true protects one JVM, not the cluster
Related
I Ship Mobile Apps for $0: Vercel + Render + Supabase Free Tier
Cloud
0
DEV Community
[Showoff Saturday] Asili - locally calculated personal DNA trait scorer and gene explorer
Cloud
0
Reddit r/webdev
Cold-Weather Photovoltaic Arrays: Calculating Sub-Zero Voc Expansion and Dielectric Breakdown in TypeScript
Cloud
1
Dev.to (EN Zone)
[Showoff Saturday] I turned a fantasy season into one visual grid (free, no signup)
Cloud
1
Reddit r/webdev
Comments0
No comments yet — be the first