Our quotes API is a Node service built by Jenkins on a pool of long lived agents. The pipeline has two stages that matter here. The test stage runs npm ci and the unit tests directly on the agent. The image stage then runs docker build with the workspace as its context, and the Dockerfile does what most Node Dockerfiles do: copy package.json and the lockfile, RUN npm ci, then COPY . . for the source.
In October we moved the image from Node 18 to Node 20. The pull request was green. The image built. In staging every pod crashed on start with an error from bcrypt: the module was compiled against a different Node.js version using NODE_MODULE_VERSION 108, and this version of Node.js requires NODE_MODULE_VERSION 115. 108 is Node 18. Nothing inside a Node 20 image should have known about it.
Our .dockerignore listed .git and the docs folder and nothing else. So the build context included the node_modules that the test stage had just installed on the agent, which still ran Node 18. Inside the image, npm ci built a correct node_modules for Node 20, and then COPY . . copied the agent's directory on top of it. COPY merges into an existing directory and replaces any file with the same name, so the native bcrypt binary from the agent overwrote the one the image had just compiled.
The uncomfortable part was not the crash. It was realising this had been happening for two years. While the agent and the image ran the same Node version, the overwrite produced identical files, and the npm ci inside the Dockerfile had been wasted work on every build. Our images contained whatever the agent had happened to install, and nobody could have told. The tests had also run only on Node 18, so the green pull request said nothing about Node 20.
node_modules is now in .dockerignore, along with coverage output and anything else a stage writes into the workspace. The test stage runs inside the same base image the Dockerfile uses, so the tests run on the Node version we ship. A check in the image stage fails the build if the context contains a node_modules directory at all.
A build context is everything in the folder, not everything in the repository, and the folder had been shaped by the stage that ran before it.
– Sergey Shinder

