At 06:53 on Tuesday 20 October 2026, an underground power cable failed outside a data centre in London. The building lost one of its two power feeds. That should have been a non-event. The site had a second feed, a bank of diesel generators, and batteries to cover the gap while the generators started.

Six hours later, an entire Google Cloud zone was still dark. Forty services were affected. Banks could not process payments properly. And the generators, which had fuel and were ready to run, never received the order to start.

The reason is the kind of bug that should make every backend developer uncomfortable, because most of us have written its smaller cousin. A sensor was measuring the wrong thing, and the system that trusted it had no way to know.

How the site was supposed to work

The facility hosts europe-west2-b, one of three availability zones in Google Cloud's London region. An availability zone is a physically separate data centre building, or part of one, with its own power and cooling, so that a failure in one zone does not take down the others.

The site used what engineers call 2N power. That means two complete, independent power paths for every critical load. If one path fails, the other carries everything. Two utility feeders, called A and B, brought power in from the grid. Behind them sat a bank of diesel generators, run by a small industrial computer called a PLC, a programmable logic controller. The PLC's job was simple: watch for a loss of grid power, start the generators, and hand the load over to them.

Between the grid failing and the generators taking over, batteries covered the gap. The UPS, the uninterruptible power supply, gave the site about five minutes of runtime. Five minutes is not much. It is exactly enough, if the PLC notices the failure and starts the generators at once.

But the site was not in its normal state that morning. Six days earlier, on 14 October, a quarterly generator load test had turned up problems on the B feeder path. The feed B breaker, the big switch connecting feeder B to the building, had been physically opened so repairs could be done. The site was running on a single feed, feeder A, carrying the whole load while B was out for maintenance. Google calls this a temporary single-feed maintenance posture. In plain words: the backup power path was already switched off when the remaining path failed.

None of that was secret or sloppy. Maintenance postures like this are normal. They are also the moment when the next failure hurts most, because the redundancy you are counting on is the thing already taken apart.

The sensor on the wrong side of the breaker

At 06:53 BST, the underground cable fault killed feeder A. At that moment, the PLC should have seen the voltage disappear and ordered the generators to start.

It did not, because of where its sensors were wired.

A breaker has two sides. The line side faces the utility grid. The load side faces the building. The PLC's undervoltage sensors were wired on the line side of the feed B breaker, upstream of the switch. Feeder B was still live on the utility side of that breaker. The breaker was open, so no power could flow from B into the building, but the sensor could not see the breaker. It could only see voltage, and voltage was present.

So the PLC asked its one question, "is there voltage on the feeder?", and got the answer "yes". Its logic did not also check the second fact that mattered: is the breaker actually closed, so that voltage can reach the building? Google's report puts it plainly. The PLC logic did not verify both an undervoltage condition and physical breaker position. Both the active and the standby controller ran identical logic and shared the same upstream sensors, so the redundancy that was supposed to protect the site had the same blind spot twice. Two controllers, one mistake.

The batteries ran out at about 07:00. Servers, storage, and network hardware in the zone lost power. When the on-site team tried a remote load transfer through the PLC, the system refused, because the same live voltage reading told it there was nothing wrong to fix. The team eventually moved the load to a standby utility source by hand, and power to the data hall came back at 08:20. But the damage was done. The main impact lasted 6 hours and 8 minutes. By 13:08, 99.5 percent of Compute Engine virtual machines were healthy again, and new virtual machines could be created. The full incident ran 6 hours and 10 minutes.

The blast radius was bigger than one zone

The physical outage was confined to zone b. The impact was not. When workloads failed over into zones a and c, the sudden demand saturated the available capacity there, and control plane calls, the calls services make to ask the platform to do things, kept timing out against components in the dead zone. The result was elevated errors and slow responses across the whole London region.

One detail deserves its own paragraph, because it is the same bug wearing a different coat. Apigee X, Google's API management service, is designed to survive losing one zone, because its datastore keeps a copy in each of the region's three zones. But a scheduling misconfiguration had placed two of the three copies in zone b. When the zone went dark, the datastore lost quorum, the minimum number of copies needed to agree before it will answer, and the service returned errors for 7 hours and 19 minutes. The design said three zones. The reality was two copies in one zone. Same shape of failure: the check that was supposed to guarantee spread was not measuring spread.

Google's full root cause went up on its status page on 4 November. It is worth reading in full. The report also lists what changed afterwards: the PLC logic was reprogrammed and retested, generator load transfer tests were run, and Google is validating PLC configurations at other sites with similar layouts. The utility cable itself was located on 26 October, with repair targeted for 13 November.

The version of this bug in your codebase

Here is why this story belongs in a backend article and not just a data centre one. The PLC's failure was a health check that checked the wrong signal. It asked "is there voltage?" when the question that mattered was "can power reach the building?" The answer to the first question was correct and useless.

Almost every service has a version of this. The most common is the health endpoint that only proves the process is alive:

func healthHandler(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) w.Write([]byte("ok")) }

This handler answers 200 while the database it depends on is down, while the disk is full, while the queue it writes to is unreachable. It is the voltage sensor on the line side of the breaker. It measures something true, and it tells you nothing about whether the service can do its job.

The fix is to check the real dependency, with a timeout so a dead dependency cannot hang the check forever:

func healthHandler(db *sql.DB) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second) defer cancel() if err := db.PingContext(ctx); err != nil { http.Error(w, "database unreachable", http.StatusServiceUnavailable) return } w.WriteHeader(http.StatusOK) w.Write([]byte("ok")) } }

PingContext opens a real connection and waits for the database to answer. Two seconds is the budget: if the database cannot answer in two seconds, the service is not healthy, and saying so is the whole point. The same shape works in Python:

@app.get("/health") def health(): try: with engine.connect() as conn: conn.execute(text("SELECT 1")) except Exception: return {"status": "unavailable"}, 503 return {"status": "ok"}

Keep the check cheap. SELECT 1 is enough. You are testing that the path is open, not running a report. And only put a dependency in the check if losing it really means the service cannot serve traffic. A health check that fails because an optional analytics sink is slow is the same mistake in reverse: a check that cries wolf, which teaches everyone to ignore it.

The second lesson is about the handover, not the backup. Google had generators. The generators worked. What failed was the handover from grid to generator, the exact step the five minute battery existed to cover. Backups fail this way all the time: the backup file exists, but nobody has ever restored from it. The replica is running, but nobody has ever failed over to it. The test that matters is not "does the backup exist" but "does the handover finish inside the buffer".

That test can be small. Here is the shape of it, as a sketch for a staging environment, not production:

func TestFailoverCompletesInsideBuffer(t *testing.T) { stopPrimary(t) // Your battery: the buffer you actually have. Five minutes // for Google, maybe thirty seconds for you. Measure it. deadline := time.Now().Add(30 * time.Second) for time.Now().Before(deadline) { if err := probeBackup(); err == nil { return // the handover worked inside the buffer } time.Sleep(100 * time.Millisecond) } t.Fatal("backup did not take over inside the buffer") }

Run it on a schedule, and run it after every change to the failover path. A handover that worked six months ago is a claim, not a fact. Google's own fix list includes exactly this: full generator load transfer tests, run after the outage, with the automatic startup checked against the battery timer.

The third lesson is about redundant pairs that share a failure mode. Two PLCs running identical logic from the same sensor are not two independent checks. They are one check with a spare. The same trap shows up in software: two database replicas in the same availability zone, like the Apigee copies. Two health checks that both call the same function. A primary and a standby that both read the same configuration file, so the bad config takes them both down. Redundancy only counts when the two sides can fail independently. If they share a sensor, a config, a zone, or a code path, you have one system with a costume change.

Three questions to take back

First, does your health check touch the real dependency, or does it measure a proxy for it? Open a connection. Run the trivial query. Check the thing, not the shadow of the thing.

Second, have you tested the handover, not just the backup? Kill the primary in staging and watch the clock. If the takeover takes longer than your buffer, your buffer is a wish, not a plan.

Third, do your redundant pairs share a failure mode? Look for the shared sensor: the same config, the same zone, the same code path, the same person who has to approve the failover. Shared failure modes are invisible until the day they are not.

The uncomfortable part of the London outage is how ordinary each piece was. A cable fault outside the building. Maintenance on the backup path. A sensor wired to the wrong side of a switch. A controller that checked one fact instead of two. None of it was exotic. That is the point. The systems that take down a whole zone are rarely exotic. They are ordinary parts, arranged so that a single wrong measurement can silence the alarm that was supposed to save them.

So here is the question to take back to your own systems: which of your health checks is measuring voltage on the wrong side of the breaker, and how would you know?

Source: Google Cloud Service Health, incident report for the europe-west2-b power disruption of 19 to 20 October 2026, updated 4 November 2026. https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj