When does npm test exit 0 even though the bug is still there?

Agent loops that fix failing tests usually stop on one signal: the test command exits with 0. It's a good signal because a model can't argue with it. The osFoundry desktop app's starter loop "Get the JS suite green" verifies each run with npm test, and a comment in its Rust source (loop_config.rs) calls a command check "the only form that cannot be talked into passing".

That holds for the model's words. It doesn't hold for the files the model can edit. The agent fixing your tests has write access to the repository, and the exit code of npm test is produced by that repository: the test files, package.json, and the command line you wrote. Below, each way an exit code can say "pass" while the bug is still there is built as a tiny Node project and run.

Thirteen cases, measured

Each case is a fresh temp folder with src/add.mjs and a node --test suite. "Bug there" means add returns a - b and the test expects add(2, 3) to be 5. The script runs the command through sh -c and records the exit code. Node 24.21.0, npm 11.19.0, macOS 15.8.1 on an M1.

Nine of the thirteen exit with 0, and only one of those nine is a real fix.

Three different failures

The rows fall into three groups, and each one needs a different guard.

The suite changed. Editing an assertion, skipping, marking todo, deleting or moving the file: in every case the exit code is accurate about the tests that ran. The tests that ran just aren't the ones you meant. The moved file is the quiet one. node --test with no arguments only picks up files matching its default patterns (*.test.mjs, anything under test/, and so on), so a test that moves out of those patterns isn't skipped, it just stops existing. The runner reports tests 0 and exits 0.

process.exit(0) at the top of a test file is the strangest row. node --test runs each file in its own process, and a file whose process exits 0 is reported as a passing test: ✔ test/add.test.mjs, pass 1. The assertion never ran.

The command lies. In npm test | tee test.log the shell returns the exit status of the last command in the pipe, which is tee, which succeeded. || true does the same on purpose. Neither is something an agent has to do. These come from the person who wrote the check, which is why the command belongs to the loop's definition, not to the agent. With set -o pipefail the same tee command exits 1 under bash.

There is no verdict. A hung test has no exit code at all, and a missing package.json gives 254, which says nothing about your code. A loop has to decide what these mean, and "fail" is not always the right answer.

Pass, fail, or not judged

The app's loop verifier has an answer for that last group. The check runs through the same execute_command tool the agents use, and command_exit_verdict in loop_verify.rs reads the last EXIT CODE: line of its output. Exit 0 passes and anything else fails. A command marked as timed out fails even if it reports 0. A result with no exit code, or a command that couldn't be started at all, is None: not judged. The comment states the reason: defaulting to pass would let a malformed result approve every run, and defaulting to fail would "burn a loop's failure budget on a reporting change".

That budget is real. The loop stops after three failed verifications in a row (DEFAULT_MAX_CONSECUTIVE_FAILURES = 3), so if an unreachable folder counted as a failure, a flaky disk would switch off a loop whose work was fine. When the command can't be judged, the app falls back to the goal ledger's own verdict for that run: met its criteria is a pass, ran out of rounds is a fail, and a downed grader, a token ceiling or a cancel are not judged at all and leave the counter where it was.

Rebuilding the judge in Python

Here is the same three-state judge as a standalone script, with guards for the first two groups added. CHECK is fixed by the loop, not chosen by the agent, and uses pipefail. CHECK_TIMEOUT is 120 seconds. Before the first round the script hashes every *.test.*js file and records how many tests the suite has.

CHECK = "set -o pipefail; npm test 2>&1" def run_check(cwd): """Pass (True), fail (False) or not judged (None). Never guess.""" try: r = subprocess.run(["bash", "-c", CHECK], cwd=cwd, capture_output=True, text=True, timeout=CHECK_TIMEOUT) except subprocess.TimeoutExpired: return False, "timed out" # a hang is a failure, whatever it prints  except OSError as e: # could not even start: no folder, no shell  return None, f"could not run: {e.strerror}" return r.returncode == 0, r.stdout def judge(cwd, baseline_files, baseline_tests): ok, out = run_check(cwd) if ok is None: return None, out if test_files(cwd) != baseline_files: return False, "test files changed" s = summary(out) if ok and (s.get("tests", 0) < baseline_tests or s.get("skipped", 0) or s.get("todo", 0)): return False, f"exit 0 but {s}" return ok, f"exit {'0' if ok else 'non-zero'} {s}" 

summary reads the counters node --test prints at the end (tests, pass, fail, skipped, todo), and test_files maps each test file's path to its SHA-256. Changing a test file is always a failure here. That matches the app's goal text for this loop: if a test is genuinely wrong, the agent should say so and stop, which in practice means a person looks at it.

The loop around it is short. At most 4 rounds, stop after 3 failures in a row, and a None doesn't touch the failure counter:

for rnd in range(1, MAX_ROUNDS + 1): agent_round(cwd, rnd) verdict, why = judge(cwd, baseline_files, baseline_tests) if verdict is True: return "verified", rnd if verdict is False: failures += 1 if failures >= MAX_CONSECUTIVE_FAILURES: return "stopped: failed 3 in a row", rnd # None leaves the counter alone: nothing judged this round return "stopped: out of rounds", MAX_ROUNDS 

To keep it to one loop, both limits sit in the same for here, and the command runs after every round. In the app, rounds inside a run are graded against the goal's criteria by a separate grading call, the command runs once when the run ends, the round limit bounds that run, and the failure cap counts failed runs in a row before the loop stops re-arming itself.

Six stand-in agents

To test the loop without a model, agent_round is a scripted list of file edits. The project has three bugs and three tests (add, slug, total of nothing). One full run of the script printed:

The fourth row is the one the hash check alone would miss. No test file changed, but package.json now points the runner at a glob with no matches, so the suite exits 0 with zero tests. The test-count guard catches it.

The last row is the three-state rule doing its job. Round 2's fix happened, but nothing could check it, so that round was neither a pass nor a strike. Round 3 then passed with all three tests.

In that run each check took between 0.17 s and 1.29 s on this tiny project, and every scenario, three or four checks included, finished in under 4 s in each of three runs. A model round costs far more than that, so there's no reason to skip the check or make it lighter.

What these guards don't cover

The guards are files and settings too. An agent that can edit package.json can also add a --test-skip-pattern flag or stub the module under test, and there will always be another way around. The more durable fix is where the check runs: from a copy of the test files the agent can't write to, or against git diff --quiet -- test/ on a clean checkout, with the check command defined in the loop and not in the repository.

The full scripts (exitcodes.py for the table, loop.py for the loop) are standard-library Python and only need Node on the PATH. In the osFoundry app, the verifier described above runs on your machine through the same command tool as the agents, which is also why it only runs when the loop's tool scope allows commands. The app is free from the download page.