The cloud agent opens its pull request with a tidy summary: all tests pass. You pull the branch, run the suite on your laptop, and the integration folder goes red.

Nobody lied. The agent's machine had no route to the staging auth service that lives behind your VPN. The fixture that needs it called pytest.skip, and a skipped test is not a failed one. The summary was true for the tests that ran.

That's my take, and the vendors' own docs back it: a cloud agent's "tests pass" is a statement about the slice of your suite it could start. If the rest needs a private registry, an internal API or a database dump that only exists on your machine, a local agent in its own worktree runs the real thing. I build a tool for the local side, so weigh this with that in mind. Every limit below links to the vendor's page, not to my opinion.

What each cloud agent actually gets

Three vendors, three sandboxes. Here is what each one's documentation says about where the agent runs and what it can reach: GitHub's Copilot environment guide and firewall page, OpenAI's Codex cloud environment and internet access pages, and Anthropic's cloud environments page.

Copilot cloud agent Codex cloud Claude Code cloud sessions
Runs in A GitHub Actions runner, or a cloud sandbox A container from the universal image An Anthropic-managed Ubuntu 24.04 VM
Agent's network by default Firewall with a recommended allowlist Off during the agent phase Trusted: package registries, GitHub, cloud SDKs
How you add things copilot-setup-steps.yml Setup script, optional maintenance script Setup script, cached if it finishes in about 5 minutes
Secrets while the agent works Agents variables or secrets Removed before the agent phase starts Network secrets, attached by a proxy outside the VM

A few details matter for tests. Copilot's setup file must hold a single job named copilot-setup-steps, and a session on GitHub Actions has a maximum execution time of 59 minutes. Codex's setup script runs with internet access, then the agent phase starts with it off unless you enable limited or unrestricted access. And Codex removes secrets before the agent runs, so a test that reads an API key at run time won't find it. Plain environment variables do stay set for the whole task.

A test suite drawn as a row of lanterns, half of them lit inside a glass cloud, the other half dark behind a locked gate
The suite a cloud agent runs is the part of your suite its sandbox can reach.

Docker Compose isn't the gap anymore

The lazy version of this argument says cloud agents can't run your services. That's out of date, and I'd rather be fair about it.

Claude Code's cloud VM ships with docker, dockerd and docker compose, plus PostgreSQL 16 and Redis 7.0, according to its installed tools table. The docs tell you to ask Claude to run docker compose up, and to put docker compose pull in the setup script so the cached snapshot keeps the images. Copilot's setup job accepts a services key, the same one a normal Actions job uses for service containers. Codex's environment pages don't say whether Docker works inside the container, so I won't guess.

So if your Postgres comes from a public image and your seed data is a SQL file in the repo, a cloud agent can stand the stack up. One catch from Anthropic's own page: the cache is a filesystem snapshot, so "a docker compose up stack, or any other background process" doesn't survive it. Every session has to start the containers again. If your fixtures skip when a service is missing, an agent that forgets this step skips your integration tests and can still report green.

What none of them reaches on the default setup

The gap is the network, plus whatever lives only on your laptop.

Your VPN. Anthropic says it outright. Its self-hosted environments page sells self-hosting on this: sessions "run inside your network and can reach internal services, databases, and registries without exposing them to the public internet." That is the thing the hosted VM can't do. A firewall allowlist doesn't change it either. Copilot lets you allow a domain like packages.contoso.corp, but GitHub's firewall page doesn't mention private networks, and allowing a hostname doesn't give a GitHub-hosted runner a route into your office network.

Your private registry. GitHub's environment guide warns that without setup steps, Copilot "may be completely unable to download these dependencies—for example, if they are private." Setup steps fix that when the registry is on the public internet with a token. They don't when it's VPN-only.

Your seed data. Cloud sessions clone from the remote. Claude Code's cloud docs say claude --cloud clones "your current directory's GitHub remote at your current branch, not your local checkout." When it uploads a bundle instead, untracked files aren't included, and on macOS, Linux and WSL uncommitted changes to .env and key files stay on your machine (native Windows uploads them as they are). That's the right call for secrets. It also means the anonymised dump in your git-ignored seed/ folder never makes the trip.

Here is the pattern that turns all of this into a quiet green check. A fixture that skips when Postgres isn't there:

# tests/conftest.py import os import socket import pytest def _reachable(host, port): try: socket.create_connection((host, port), timeout=1).close() return True except OSError: return False @pytest.fixture def db_port(): port = int(os.environ.get("DB_PORT", "5432")) if not _reachable("localhost", port): msg = f"postgres not reachable on {port}" if os.environ.get("REQUIRE_SERVICES") == "1": pytest.fail(msg) pytest.skip(msg) return port

Skipping is friendly on a laptop with Docker stopped. In an agent's sandbox it hides the hole. The REQUIRE_SERVICES switch turns a missing service into an error, and running pytest -rs prints every skip reason in the summary. Whichever agent you use, ask it to paste that summary into the PR. A list of skipped tests tells you more than "all tests pass."

Where the cloud agent is the right call

None of this makes cloud agents a bad idea. They're the right call for a self-contained repo: a library, a CLI, a frontend with mocked APIs. If git clone plus a public install gives you the whole suite, the sandbox runs the same tests your laptop would.

They also fit work you start away from the desk. A Claude Code cloud session keeps running after you close your laptop, and you can start one from the Code tab in the Claude mobile app. claude --cloud starts several tasks that run at once in separate sessions. Copilot's cloud agent works on one branch and opens one pull request per task.

A team can also bring the sandbox inside its network, and both GitHub and Anthropic have a path for that. Copilot runs on self-hosted runners, typically managed by ARC, as long as you turn its firewall off for them, and only on Ubuntu x64 or Windows 64-bit. Claude Code's self-hosted environments, in public beta on Team and Enterprise plans, run sessions on runners you host. Anthropic's own advice is that "most teams are better served by Anthropic-hosted environments," because you then own the runner image and the fleet. For a team with a platform group, that's the real fix. For one developer on a VPN, it's a lot of infrastructure to get a test suite to run.

The local version: one worktree, its own ports

On your machine, the agent sits on the same network you do. The private registry resolves, the staging API answers, the seed dump is right there. The problem locally is the opposite one: two agents on one checkout, both trying to bind Postgres to port 5432.

The fix is a worktree per task and a port per worktree. Docker Compose helps here on its own: the default project name is "the base name of the project directory," and Compose uses that name "to isolate environments from each other." Two worktree folders give you two sets of containers. You still need different host ports, so publish one from a variable:

# compose.yaml services: db: image: postgres:16 environment: POSTGRES_PASSWORD: app ports: - "${DB_PORT:-5432}:5432" healthcheck: test: ["CMD", "pg_isready", "-U", "postgres"] interval: 2s retries: 15

I build Hermes IDE, which does the per-task part. ⌘N (Ctrl+Shift+N on Windows and Linux) starts each task on its own hermes/<name> branch in its own worktree, with the task as the agent's first prompt. Claude Code, Codex, Copilot CLI and the rest each run in their own real terminal, signed in with your own accounts.

Two agent sessions side by side, Codex and Claude Code, each in its own hermes/ worktree, with the sessions sidebar on the left
Codex on orders-types and Claude Code on orders-api, each in its own hermes/ worktree, with status, context use, cost and branch in the sidebar.

A worktree recipe prepares each new worktree before the agent starts. For the stack above (macOS and Linux, since setup runs in sh):

# .hermes/worktree.toml copy = [".env", "seed/dev.sql"] setup = [ "echo DB_PORT=$HERMES_PORT_DB > .env.ports", "docker compose --env-file .env.ports up --wait", "docker compose exec -T db psql -U postgres < seed/dev.sql", "pip install -r requirements.txt", ] done_when = ["sh scripts/test-all.sh"] [ports] db = 5432

[ports] gives each worktree the first free port at or above 5432 that no other worktree holds, as HERMES_PORT_DB. copy brings over git-ignored files only. If a pattern matches a tracked file, Hermes refuses it with a message and setup doesn't run, so the seed dump has to stay in .gitignore. Ignore .env.ports too, since setup writes a different port into it in every worktree. up --wait waits for the healthcheck. The first time Hermes sees the recipe, and whenever it changes, it shows you the commands, and nothing runs until you choose Run setup.

done_when points at a script that runs the suite with missing services turned into failures. Other skips still skip, and -rs lists them:

#!/bin/sh # scripts/test-all.sh set -e set -a . ./.env.ports set +a REQUIRE_SERVICES=1 pytest -rs

The Review Desk's Worktrees tab listing one Hermes worktree each for orders-api, web-app and orders-types, with branch, task, age and disk size
The Worktrees tab: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each one's hermes/ branch, task, age, disk size and the free disk space.

The local route has its own cost. The agent reaches everything you reach. If your VPN routes to production, so does the agent's shell, so give the dev stack its own credentials and keep approvals on. Hermes never answers a permission prompt for you. It also has no hosted agents of its own, so a session on your laptop stops when the laptop sleeps. Hermes is free, source-available under BSL 1.1, runs on macOS, Windows and Linux, and isn't affiliated with Nous Research, Anthropic, OpenAI or GitHub.

So which "tests pass" do you trust?

My rule: if a fresh git clone on a public network can run the whole suite, send the task to the cloud and enjoy the parallelism. If it can't, either bring the sandbox into your network or run the agent where the network already is. And whichever you pick, make missing services fail loudly.

What does your suite need that a cloud sandbox can't reach? And has an agent ever handed you a green check that turned out to be a pile of skips? Tell me in the comments.

If this was useful

Getting a test suite to run the same way on a laptop, in CI and in an agent's sandbox is part of the plumbing between localhost and production. Ship It covers that plumbing for a small team: hosting, deploys, CI/CD, auth, payments and monitoring, with the trade-offs spelled out. If you want to try the local side of this post, Hermes IDE is a free download, and the repo is on GitHub.

Ship It — the pragmatic startup tech stack