The setup was deliberately boring. A small Node.js app with SQLite on a single VM, a release-directory layout with a current symlink, a CI workflow that uploads each release over SSH, runs npm ci, flips the symlink, restarts the systemd unit and waits for /healthz. If the health check fails, flip the symlink back and restart again.

Every piece of that looked right in review. The first time CI ran it end to end, the deploy went red, the app went down, and the rollback — the part that exists precisely for this moment — failed as well.

Neither failure was in the deploy logic. Both were in what the tools around it did by default.

TL;DR: rsync -a includes -p and -g, so it copies the sender's mode and group onto the target — including onto the release directory itself, which silently clears the setgid bit the server relied on. npm ci then wrote node_modules with the deploy user's private group, and the app, running as a different user that only shares a group, couldn't read its own dependencies. The app crash-looped into systemd's StartLimitBurst, after which systemd refused every start — including the rollback's — until systemctl reset-failed. Fix: normalise group and setgid after the upload, call reset-failed before every restart, and stop waiting on a health check once the unit has visibly crashed.

Why Two Users in the First Place?

Because the process that writes code to the server shouldn't be the process that runs it.

The app runs as an unprivileged runtime user with no sudo and no SSH key. Deploys run as a separate deploy user whose key is restricted and whose sudo rights cover exactly two commands: restarting the unit and reset-failed (why the second one is on the list is most of this post). The two share one group, and the shared directories are set up so both can write:

# once, at setup chgrp -R app-shared /srv/app chmod -R g+rwX /srv/app find /srv/app -type d -exec chmod g+s {} + # new files inherit the group

The setgid bit on a directory is what makes this work without either user having to think about it: every file created inside inherits the directory's group instead of the creator's primary group. Combined with umask 0007 on both sides, everything the deploy user creates is group-readable and group-writable, and everything outside the group sees nothing.

That's the property the first deploy lost.

What Did rsync Actually Change?

The workflow creates the release directory first, then uploads into it:

ssh deploy-target "mkdir -p '/srv/app/releases/${REL}'" rsync -az ./ "deploy-target:/srv/app/releases/${REL}/"

The mkdir happens inside releases/, which has setgid, so the new directory is created correctly: shared group, setgid set. Then rsync runs, and -a is shorthand for -rlptgoD. Two of those letters do the damage:

  • -p copies permissions from the sender. On the CI runner, the checkout directory has no setgid bit — why would it. A trailing slash on the source (./) means rsync syncs the attributes of the directory itself as well as its contents, so it chmods the freshly created release directory to the runner's mode. setgid gone.
  • -g copies the group. A non-root receiver can only assign groups it is a member of, so rsync falls back to the deploy user's own primary group for everything it creates.

Nothing about this looks wrong in a listing unless you're looking for it. The files are there, the sizes match, the owner is the deploy user as expected. The difference is one letter in the mode column and one name in the group column.

The next step made it fatal. npm ci runs on the server, as the deploy user, inside a release directory that no longer has setgid. Every file in node_modules got the deploy user's private group, with umask 0007 stripping all "other" bits. The runtime user — different UID, member only of the shared group — couldn't read a single dependency. require('express') failed on the first line of server.js.

Why Didn't the Rollback Save It?

Because by the time the script tried to roll back, systemd had stopped taking orders.

The unit had sensible crash-loop protection:

[Unit] StartLimitIntervalSec=300 StartLimitBurst=5 [Service] Restart=on-failure RestartSec=5

Crash, wait five seconds, restart, crash again. Five starts inside five minutes and systemd marks the unit failed with start request repeated too quickly. That's exactly the right behaviour for a broken binary at 3 a.m. — it stops a crash loop from hammering the machine.

What's less obvious is that the start limit doesn't distinguish who's asking. Once it trips, a manual systemctl restart is refused just like the automatic ones. The deploy script flipped current back to the previous, known-good release and called restart — and systemd rejected it, because the limit belongs to the unit, not to the code the unit happens to point at.

So the sequence was: new release can't read its dependencies, unit crash-loops through its burst, health check eventually times out, rollback flips the symlink correctly and then can't start anything. The old release was sitting on disk, intact, one symlink away, and the service stayed down until someone ran reset-failed by hand.

What Does the Fix Look Like?

Three changes, each small, each aimed at one of the mechanisms rather than at the symptom.

Normalise the release after upload, before anything runs in it. The group to use is read from releases/ itself, so the script doesn't hard-code a group name:

GRP="$(stat -c %g "$BASE/releases")" find "$REL_DIR" -exec chgrp "$GRP" {} + find "$REL_DIR" -type d -exec chmod g+rwxs {} +

This runs before npm ci, so node_modules is created under a setgid directory again and inherits the right group. An alternative is to change the rsync flags — --no-p --no-g --chmod=Dg+s,ug+rw — and that works, but it puts the correctness of the server layout into a CI file that lives somewhere else. The find after the fact is idempotent and lives next to the thing it protects.

Reset the start limit before every restart, not just before rollbacks:

restart_service() { sudo -n systemctl reset-failed "$SERVICE" 2>/dev/null || true sudo -n systemctl restart "$SERVICE" }

reset-failed on a healthy unit is a no-op, so there's no reason to make it conditional. It's also why the deploy user's sudo rule has to name it explicitly — a rule that only allows restart reproduces the original failure with a permission error on top.

Stop waiting once the unit has visibly crashed. The original health check polled /healthz ten times, three seconds apart, plus curl timeouts — up to about 35 seconds of a dead app behind the reverse proxy returning 502. But there's no point polling an HTTP endpoint on a process systemd has already reported as dead:

healthy() { for i in $(seq 1 "$HEALTH_RETRIES"); do curl -fsS --max-time 5 "http://127.0.0.1:${PORT}/healthz" >/dev/null 2>&1 && return 0 n="$(systemctl show -p NRestarts --value "$SERVICE" 2>/dev/null || true)" if [ "$(systemctl is-active "$SERVICE")" = "failed" ] || [ "${n:-0}" -ge 1 ]; then log "service crashed - not waiting for the health check any longer" return 1 fi sleep 3 done return 1 }

A unit that has restarted even once since the deploy's own restart is not going to come good on the next poll. Failing fast here shortens the outage of a bad release from half a minute to a few seconds, and gets to the rollback while the start limit still has budget left.

Couldn't This Be Caught Before It Reaches Production at All?

Yes — and that became the next change. The three fixes above make a broken release fail faster and recover reliably. They don't stop a broken release from being switched in. The deploy now boots every release against a copy of the production database on a spare port before touching current, and the very first thing that preflight does is look for exactly the failure from this post: files in the release that the shared group can't read.

bad="$(find "$REL_DIR" \( -type d ! -perm -g=rx -o -type f ! -perm -g=r \) | head -n 3)" [ -z "$bad" ] || { log "not group-readable: $bad"; return 1; }

That check exists because the preflight itself runs as the deploy user, not the runtime user — so a permissions problem would sail straight through a "does it boot" test. The rest of that design is the next post.

What's the Generalizable Lesson Here?

  • rsync -a is "archive", and an archive preserves the sender's metadata. When the receiver's metadata is the part that matters — shared groups, setgid, ACLs — either strip -p/-g explicitly or normalise afterwards. A trailing slash on the source applies the source directory's attributes to the target directory too.
  • setgid on directories is a quiet invariant: nothing breaks when it's removed, until the next file is created. Anything that relies on it should re-assert it, not assume it.
  • With separate deploy and runtime users, "it works for me" during deploy proves nothing about the runtime user. Check readability as the group, not as yourself.
  • systemd's start limit is per unit and blocks manual starts as well as automatic ones. Any script that restarts a unit after a possible crash loop — a rollback above all — needs reset-failed first, and the sudo rule needs to permit it.
  • Don't keep polling an HTTP health endpoint on a process the supervisor has already declared dead. Ask the supervisor first; it knows sooner.
  • A rollback that has never run against a real failure is a hypothesis. The interesting state isn't "previous release on disk" — it's "unit just crash-looped through its start limit", and that's the one to rehearse.