Our new Compose project took a carrier's address range and the host stopped reaching them

Our integration host is a virtual machine in our data centre that runs one Docker Compose project per partner adapter: carriers, two marketplaces, our payment provider's settlement files. In October it ran eleven of them. One carrier, which we reach through a site to site VPN, sits at 172.25.10.0/24, and our firewall routes that range into the tunnel. The host knows nothing about the VPN. It sends everything that is not local to its default gateway.

On a Tuesday afternoon we deployed a twelfth adapter, for a new returns partner. Twenty minutes later the carrier adapter was logging connection timeouts on every booking, and the firewall showed no traffic for the tunnel at all.

Every Compose project gets its own bridge network unless told otherwise, and Docker chooses its subnet from a default list of pools: 172.17.0.0/16 for the default bridge, then 172.18.0.0/16 through 172.31.0.0/16, then slices of 192.168.0.0/16. It takes the first one that no other Docker network on the host is using. It has no way of knowing about networks beyond the host, and the carrier's sat behind our default gateway. Our new project was given 172.25.0.0/16. The host gained a connected route for that range on the new bridge, more specific than the default route, and from then on every packet for the carrier went into an empty bridge on the same machine.

The range had been free for years only because projects had come and gone and the count had never climbed that far. Nobody on the host or at the carrier had changed anything that touched the carrier. The pool had simply filled up as far as them.

We stopped the new project, which removed its network and the route with it, and bookings went through within a minute. Then we set default-address-pools in daemon.json to a block our network team reserves for container hosts, 10.212.0.0/16 carved into /24 networks, and set bip so the default bridge comes from the same block. Existing networks keep their subnets until they are recreated, so we brought each project down and up again on a quiet evening. A check now compares every bridge subnet on the host with the partner ranges in the firewall configuration, and runs before any project starts.

Docker picked a range that nobody on that machine was using. The machine was the only place it could look.

– Sergey Shinder