Shipkeep is an open-source platform to deploy apps on your own servers, where the web panel never holds an SSH key. Each server runs an agent that connects out over mTLS and only accepts a closed list of typed actions.
Until this week, no deployment had ever gone past queued outside of tests. Then I ran the whole thing end to end for the first time: docker compose up for the control plane, the agent installer on Ubuntu 24.04, then a deployment.
About 600 unit tests were green. It took five fixes before one container served one request.
None of the five was reachable by the tests I had, and for the same reason each time: each side was tested against a stand-in for the other.
1. The agent said "done" before it said what it had done
The agent journals every action and reports each state change. The control plane closes an action on the first final update it receives.
The agent sent "succeeded" from its journal notification, without the result. Then it sent a second "succeeded", this time with the container id. That second message arrived for an action that was already closed, and was dropped.
So the container was created, the control plane concluded The agent returned no container id, failed the deployment and removed the container. Every deployment, every time.
The agent's test checked its last update. The control plane's test fed it a single update. Both were right about their own side. The test I should have had from the start:
// The control plane closes the action on the first final update: it must hold the result. for _, sent := range updates { if sent.GetState() == SUCCEEDED && sent.GetResult().GetContainerId() == "" { t.Fatalf("a succeeded update was sent without its result: %v", sent) } } 2. My fake Caddy was more polite than Caddy
The agent creates its HTTP server in Caddy when it is missing, by walking up the config path until it finds something that exists. It expected 404 for a missing path.
Real Caddy answers a read below a missing parent with 400 invalid traversal path. My test fake answered 404, because that is what I assumed Caddy did.
On a freshly installed server, Caddy's config holds nothing but its admin endpoint. So the walk failed on the first step, and no route could ever be added.
The fix is small: a 400 on a read counts as "missing", only during that walk. The more useful fix is that the fake now answers the way Caddy does.
3. Restarting the control plane took the site down
In the compose file, Caddy shared the control plane's network namespace, so that it could reach the API on loopback:
caddy: network_mode: service:control-plane A container that joins another one's namespace loses it when that one restarts. One restart of the API and the site answered nothing, until someone restarted Caddy by hand.
Now both join a third service that only holds the namespace and the published ports, and has no reason to restart:
net: command: ['node', '-e', 'setInterval(() => {}, 2147483647)'] ports: ['80:80', '443:443', '7443:7443'] control-plane: network_mode: service:net caddy: network_mode: service:net No unit test finds this one. Restarting things on a running install does.
4. An empty variable is not an unset variable
SMTP credentials are optional. A compose file cannot omit a variable, it can only leave it empty, so I made the config schema read "empty" as "unset". With a unit test, which passed.
In the running app, NestJS's ConfigService falls back to the raw environment for a key that validation left undefined. It handed back the empty string. Every email then failed with expose is not a function, inside a handler that hides SMTP errors on purpose, so that server responses and credentials are never stored.
Sign-up needs a confirmation email. Nobody could create an account, and nothing said why.
The unit test called the parser. The app calls the service.
5. docker.io/traefik/whoami is not traefik/whoami. Except when it is.
Two bugs behind one symptom.
First, docker.io does not serve the registry API: it redirects, and the API lives on registry-1.docker.io. Resolving the image digest failed.
Second, once resolved, the agent pulled docker.io/traefik/whoami@sha256:… and checked that the local image carried exactly that name. Docker records it as traefik/whoami. The pull succeeded and was reported as failed.
Then the fix had a hole of its own, which a code review caught. Dropping the host from docker.io/127.0.0.1/app gives 127.0.0.1/app, and Docker reads that as a registry on localhost. That goes straight past the check that keeps image pulls away from private addresses. Those references are now refused.
The shortcut that cost a day
When bug 5 first showed up, the run switched to the short name, traefik/whoami, to keep going. A fix was written with unit tests, and merged. Nobody redeployed the docker.io/… form that had actually failed.
The next day, on a clean run from scratch, it failed again, one step further.
I build Shipkeep with an AI coding assistant, and that shortcut was its call. It is also the most human shortcut there is: swap in the input that works, and promise to come back. The rule is now written down for both of us:
When something fails, fix the cause, then rerun the exact case that failed, in the place where it failed, before anything else moves.
Where it stands
A first prerelease is out: v0.1.0-alpha.1. The control plane image is signed and built for amd64 and arm64, next to the agent and CLI binaries.
| State | What |
|---|---|
| Ran with the published release, on one machine with Docker | Control plane, sign-up, a server installed with the command from the Servers page, deployments, a rollback, a restart of the server's container |
| Not run yet | A real server or VM and its reboot, a publicly trusted certificate, a real mail provider, arm64 at run time, an upgrade between two releases |
| Not built yet | Git providers and builds, multi-server networking and failover, managed databases, backups |
The "server" in that run was a privileged Ubuntu 24.04 container with systemd, not a real machine. The installation guide lists exactly what was tested, and what was not.
A security review of the code also found five issues. Three are fixed, one of them only in part: a quota on agent enrollment is still missing. Two are open because they need a decision: no CPU or memory limit by default, and a removed domain released while its route still serves.
Where you can help
- You have a disposable Ubuntu 24.04 VM, a domain name and an SMTP account: follow the installation guide and tell me where it breaks. After this week, I expect it to break somewhere. That report is the most useful contribution right now.
- You have an opinion on defaults: the two open issues above need one.
- First contribution: the good first issues each say what to change and when it is done.
Use the prerelease on servers you can afford to lose.
And a question for the comments: what is the bug your test doubles hid from you the longest?
