DevOps
The shell step that could not fail no matter what we did to it
Sergey Shinder DEV Community
2 views
Our integration suite had been green for eleven weeks. That is not a boast, it is the symptom. A colleague onboarding to the repo wrote a deliberately broken test to see what a failure looked like, pushed it, and the pipeline went green in the usual four minutes.
The stage was a single shell step, npm run test:integration | tee integration.log || true, and it contains two separate mistakes, both added by people trying to be helpful. The pipe was there so the log could be uploaded as an artifact, and without set -o pipefail the exit status of the pipeline is the exit status of tee, which is always zero. The || true had been added fourteen months earlier because a cleanup command at the end of the script occasionally returned 1 and nobody wanted to debug it that week. Either one alone would have hidden every failure. We had both.
I went looking for the rest and found seven more steps across four repositories that could not report failure: three with the same pipe, two with || true, one that ran the real command inside a subshell whose status was discarded, and one where a for loop over services swallowed each iteration's result and exited on the status of the loop itself.
The fixes were mechanical. Every script step now starts set -euo pipefail, enforced by a lint job that reads the workflow YAML and fails on any run: block that does not. || true is banned outside a small allowlist, and each allowed use carries the reason in a comment on the same line. Logs are captured by redirecting to a file and cating it afterwards, not by piping the command that matters.
The part I would keep if I could only keep one thing is the canary. There is now a job in every pipeline that runs a command which exits 1, and the pipeline asserts that this job is marked failed. It runs on every build and costs two seconds. It would have caught this in an afternoon rather than in eleven weeks.
A test suite you have never watched go red is not evidence about your code. It is evidence that a script exited zero.
– Sergey Shinder
Read original: https://dev.to/sergey_shinder_ab2d943365/the-shell-step-that-could-not-fail-no-matter-what-we-did-to-it-f45
← Previous
prefill and default look the same in Console. Only one of them lets an agent skip your required field.
Next →
Two Sources of Truth Will Always Disagree
Related
Platform Engineering: Mehr als nur DevOps? Der ultimative Guide
DevOps
0
Dev.to (EN Zone)
The Mainframe Modernization Crossroads: Evaluating Proprietary GenAI Abstraction vs. Open DevOps Architecture
DevOps
4
Dev.to (EN Zone)
How to prepare your website for Black Friday: the 12-week plan
DevOps
4
Dev.to (EN Zone)
Why AI Agents Shouldn’t Guess at Vulnerability Exploitability
DevOps
6
DevOps.com
Comments0
No comments yet — be the first