The "distroless" container that had a shell all along

For four months my docker-compose.yml carried a comment like this one:

proof-server: # No healthcheck: the proof-server image is distroless (nix-built) and has  # no shell or curl/wget binaries inside, so `docker compose --wait` cannot  # gate on it. The deploy script polls the proof-server from the host side  # before submitting transactions.  image: midnightntwrk/proof-server:8.0.3 

Every clause of that comment was wrong except "no curl/wget". The image is not distroless, and it does have a shell: bash, sh, and a full set of coreutils. I had never checked — I had inferred it from the fact that ls doesn't work inside.

That inference made --wait meaningless, and the workaround only covered one deployment path: a polling loop in the deploy script that any other entry point — a second script, a CI job, a teammate's first run — would not inherit.

What the wrong assumption looks like when it runs

Measured on the actual image, same host, same compose file:

Compose configuration up --wait returns after Service ready?
no healthcheck 2–4 s No — /health still refusing connections six seconds later
with healthcheck 13 s Yes — Up (healthy), /health returns 200 on the next request

A couple of seconds is the container starting, not the server starting. Any script that trusted --wait and immediately submitted a transaction was racing a process that needs roughly thirteen seconds before it answers.

Ten seconds of checking

The claim was "no shell". The test is one line:

docker run --rm --entrypoint /bin/sh midnightntwrk/proof-server:8.0.3 -c 'echo shell-ok' # shell-ok 

/bin is not empty either:

docker run --rm --entrypoint /bin/bash midnightntwrk/proof-server:8.0.3 \ -c 'echo /bin/*' # /bin/[ /bin/base64 /bin/bash /bin/cat /bin/cp /bin/date /bin/df ... /bin/sh ... 

So why did ls fail inside the container and make the image look distroless?

docker run --rm --entrypoint /bin/bash midnightntwrk/proof-server:8.0.3 \ -c 'echo $PATH' # /nix/store/k4a492bl82lg24l8vr6f0mcb77f5hfai-ledger-8.0.3/bin 

PATH contains exactly one nix store directory — the ledger binaries. Not /bin, not /usr/bin. Typing ls fails with command not found, which is indistinguishable from "there is no ls" unless you look. A minimal PATH is not a minimal image.

What the image genuinely lacks is an HTTP client: no curl, no wget, no nc. That part of the comment was right, and it's the part that matters least — because bash can open a TCP socket by itself.

The healthcheck that works

services: proof-server: image: midnightntwrk/proof-server:8.0.3 environment: PORT: 6300 ports: ["6300:6300"] healthcheck: test: ["CMD", "/bin/bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/6300"] interval: 3s timeout: 3s retries: 20 start_period: 15s 

/dev/tcp is a bash builtin, not a binary, so it needs no packages and no PATH. The check opens a socket and exits 0 if the kernel connects, non-zero if it doesn't. start_period is generous because the first twelve seconds are the server loading its proving keys — failed checks in that window shouldn't count against the retry budget.

With that in place, docker compose up --wait blocks for about thirteen seconds and returns only once the service is actually listening. The application-side poll stays as a second line of defence for non-compose deployments, but it is no longer the only thing standing between a deploy script and a refused connection.

Two notes if you're adapting this:

  • The check is reachability, not correctness. A TCP connect proves the port is bound. It does not prove the server can produce proofs. For that you need a real request — which brings its own trap, because reading an HTTP response from bash can block on a kept-alive connection and get your healthcheck killed by its timeout.
  • If you write the HTTP variant in compose, escape $. Compose interpolates the file before the container sees it, so $line becomes an empty string and the check silently fails. Write $$line.

When the image really has nothing

Some images genuinely ship no shell: gcr.io/distroless/static, scratch builds, most Go binaries. For those, you have three options, in order of preference:

  1. Have the application expose health in a way the runtime can see — a /health endpoint plus an external checker. The container doesn't need a shell; something outside it does.
  2. Add a healthcheck sidecar — a small image with curl that shares the network namespace, run as its own service.
  3. Check from the orchestrator — a Kubernetes httpGet probe, or a host-side poll like the one I originally wrote, wired into every deployment path rather than one script.

What you should not do is what I did: assume the image can't be checked, skip the healthcheck, and let --wait mean "the container was created".

The rule

Distroless is a specific claim — no shell, no package manager, only the app and its runtime dependencies. It is checkable in one command, and it is worth checking before you build a workaround around it.

The failure mode here wasn't exotic. It was a PATH with one entry, a command that returned not found, and an assumption that survived four months because nothing forced me to test it. The healthcheck was one line the whole time.