My circuit breaker let 100 requests through: lessons from building a small load balancer in Go

I built loadbear, a small HTTP load balancer / reverse proxy in Go, mainly to see how concurrency and resilience patterns fit together in one program. It has four balancing strategies, active health checks, a circuit breaker, retries, per-IP rate limiting and an admin port with Prometheus-style metrics. It is not meant to replace nginx or HAProxy.

After the first release, a reviewer asked a question that turned into the most useful week of the project: "How is half-open admission bounded?" Here is what we found, what was wrong, and what the numbers say.

Repo: loadbear

1. The half-open state let everything through

A circuit breaker has three states. When the cooldown ends, it moves from open to half-open and should let one probe request through to test the backend. If 100 requests arrive at that moment, all 100 shouldn't hit a backend that just recovered.

I wrote a test with 100 concurrent goroutines. All 100 were admitted. The check looked at the state, but nothing limited how many requests could pass it.

The fix is to make admission a single atomic operation. A simplified sketch of the idea (not the exact code):

type breaker struct { state atomic.Int32 // closed, open, halfOpen probing atomic.Bool // is a probe in flight? } func (b *breaker) allow() bool { switch b.state.Load() { case closed: return true case halfOpen: // only the goroutine that flips false -> true gets to probe return b.probing.CompareAndSwap(false, true) default: return false } } 

The less obvious part is the other direction: the probe slot has to be released on every exit path, including timeouts, client cancellation and panics. Otherwise a cancelled probe could leave the breaker stuck in half-open. I added a regression test for that and a deferred release.

2. Retries were less safe than they looked

I had limited retries to GET, HEAD and OPTIONS, which felt safe. Review and tests found more cases:

• A GET with a body that cannot be replayed was still retried.
• A retry could start after the client's deadline had already passed, because backoff did not look at the request context.
• Once response headers have been sent to the client, retrying is impossible, and the code needed to say so explicitly.

Retries now require a replayable body, an uncommitted response and the original request context. I also added fault modes to the fake backend (hang, close, drop-mid, slow), so these cases can be reproduced on demand.

3. WebSocket "didn't work" and SIGHUP was broken

The README said WebSocket upgrades were unverified. When I wrote a test, it failed immediately with a 400:

• The proxy stripped Connection and Upgrade from every request as hop-by-hop headers.
• My access-log response wrapper did not implement http.Hijacker or http.Flusher, so ReverseProxy could not set up the tunnel.

Wrapping a ResponseWriter is an easy way to silently lose interfaces. After the fix, an HTTP/1.1 upgrade handshake and a bidirectional byte echo work through the proxy. I only verified that, not full RFC 6455 behavior, and the README says so.

While adding config reload, I found that SIGHUP was wired into the shutdown signals. In v0.1.x, sending SIGHUP killed the process instead of reloading it. It now reloads the backend list and strategy, keeps state for unchanged backends, rejects invalid configs and lets in-flight requests finish. Only that part reloads; other settings still need a restart.

4. Benchmarks, and a measurement bug in my own tool

I wrote a small load generator (standard library only) so the benchmark works on Windows too. The first run reported errors for roughly one request per worker. The proxy was fine: my generator cancelled in-flight requests at the end of the measured window and counted them as failures. The fix was to stop starting requests at the cutoff and let in-flight ones drain.

Final results on a Ryzen 5 7600 (Windows 11, Go 1.27.4, 10 s measured, concurrency 64, rate limiting off, access logs off, client, proxy and fake backends on the same machine):

Going through the proxy costs roughly 3.5x in throughput and about 1.5 ms of median latency on this setup. The three-backend numbers are not evidence of scaling, since everything shares the same CPU. The four strategies were within run-to-run variance, so I make no claim about which is faster. I did not compare against nginx or HAProxy.

What I took away

• A test that fires 100 goroutines at one decision point finds bugs that reading the code does not.
• If something is "unverified" in your README, write the test. Two of my three "known limitations" turned out to be real bugs.
• Wrapped ResponseWriters need to keep Hijacker and Flusher.
• Measure your measuring tool before you trust its numbers.

A note on how this was built: I used GitHub Copilot to write a large part of the code and tests, so I'm not going to pretend I typed every line myself. What I did was run everything, read the failures, try the failover by hand, and follow up on a reviewer's questions. That's also why a second pair of human eyes on the concurrency code is something I genuinely want.

Feedback welcome

Failure handling is the part I'd most like more eyes on: the breaker, retries and health checks. There are open good first issue tasks in the repo if you want to contribute.

loadbear