On Wednesday 28 October 2026, the RIPE Database started failing updates. Not all of them. Some went through fine. Others were refused. The same service, the same hour, the same kind of request, and nobody could tell from the outside which ones would fail.
That is the worst kind of outage. A service that is fully down is easy to understand. A service that works for some people and not others, with no pattern you can see, makes everyone doubt their own code first.
The people who run the RIPE Database are the RIPE NCC, the registry that hands out IP addresses across Europe, the Middle East and parts of Central Asia. Their database, usually called Whois, is the record of who holds which block of addresses. If you want to change who owns a block, or set up reverse DNS for it, you send an update to this database. Reverse DNS is the record that maps an IP address back to a name, and mail servers use it to check that the server sending them email is who it claims to be. So when updates to this database fail, address records stop changing, and reverse DNS changes stop going through.
The failure ran from 11:42 to 19:40 UTC, just under eight hours, and it was intermittent the whole time. The postmortem, published on 3 November, is short. Three sentences of cause, and they are worth reading slowly, because the bug is not where you would expect it.
The clue that did not fit
Start with what the operators could see. Updates were failing intermittently. Not every update, and not in a steady pattern. If the database had been down, everything would have failed. If the update service had been down, everything would have failed. Intermittent failure, spread across eight hours, points somewhere else: something in the path between the application and the database was different for different requests.
The postmortem gives the answer in one sentence. There had been a short interruption on the network link to the primary load balancer, and the system failed over to the standby load balancer. Because of a misconfiguration, new Whois update connections that went through the newly elected load balancer were sent to the read-only database instead of the primary database. Existing connections from the application stayed connected to the primary database and were unaffected.
Read that again, because it contains the whole story. Two groups of connections, two different destinations, one working and one not.
A short glossary, because the jargon hides the mechanism
A load balancer is a machine whose job is to stand in front of a service and hand each incoming connection to a server behind it. You run two of them, a primary and a standby, so that if one dies the other takes over. That takeover is called a failover, and it is supposed to be the boring part of the system, the part that works without anyone thinking about it.
A database primary is the one copy that accepts writes. A read-only replica is a copy that keeps itself up to date by copying from the primary, and exists so that read queries can be spread across machines without loading the primary. It can answer questions. It cannot accept changes. Sending a write to a read-only replica is like posting a letter to a library: the building is real, the address is correct, and nothing you send there will ever be filed.
So the failover did its job. Traffic moved to the standby balancer. The standby balancer then sent new connections to the wrong database. And the old connections, the ones opened before the failover, were still sitting on the primary, happily accepting writes. That is why the failure was intermittent. It depended on when your connection was opened, and nobody looking at the service from the outside could see that.
Why the old connections masked the fault
This is the part that should worry you, because it is the part that repeats in other systems.
Every check the operators had would have looked reassuring. The standby balancer was up and answering. The database was up and answering. The application was connected and processing some updates. If your monitoring asks "is the balancer alive" and "is the database alive", both answers were yes. The system was green on every dashboard that measured liveness, and broken on the one thing that mattered: could a new update actually land?
The old connections made it worse. They were proof, visible to anyone watching, that the system worked. An operator seeing successful updates would reasonably conclude the problem was elsewhere, maybe with the people submitting the updates, maybe with a bad batch of requests. The healthy connections were not just unaffected. They were camouflage.
There is a second quiet detail in the postmortem. On 30 October, two days after the incident, RIPE NCC reported that none of the failed requests would have changed a reverse delegation anyway, because they either contained no changes or failed their pre-delegation checks. That is good news for the internet, and it is worth stating plainly: no valid reverse DNS change was lost, as far as the published evidence shows. It is also luck. The mechanism did not care what was in the requests. The next time a misrouted write path fails, the requests it drops might be the ones that matter.
RIPE NCC has not published how many updates failed, or which configuration setting was wrong, or a trace through the standby path. So this article does not guess at those numbers. The mechanism is enough.
The test that would have caught it
Here is the uncomfortable question. The standby balancer existed for exactly this moment, and the misconfiguration in its path had presumably been sitting there since it was set up. How do you catch a fault in a path that only runs during a failover?
You test the failover the way the system will actually use it: with a write.
Most health checks do not do this. A typical database health check opens a connection and runs SELECT 1, which is a read. A read succeeds on a read-only replica. A read succeeds on a primary. It tells you the database is alive and says nothing about whether it can accept a write. It is the exact check that would have passed on both sides of this incident.
A write-path check is different. It writes a small probe row to a scratch table, reads it back, and deletes it. If the connection is pointed at a read-only replica, the write fails at once, and you learn the truth in milliseconds. Here is the idea in Go, using the standard database package and Postgres:
// ProbeWrite checks that this connection can actually write, // not just that the database answers. Run it against the // connection your application will use for updates, including // after a failover. func ProbeWrite(ctx context.Context, db *sql.DB) error { _, err := db.ExecContext(ctx, `INSERT INTO write_probe (id, checked_at) VALUES (1, now()) ON CONFLICT (id) DO UPDATE SET checked_at = now()`) if err != nil { return fmt.Errorf("write probe failed: %w", err) } return nil } And the same check in Python:
def probe_write(conn): """Fail fast if this connection cannot write. A read-only replica will reject the INSERT at once.""" with conn.cursor() as cur: cur.execute( "INSERT INTO write_probe (id, checked_at) VALUES (1, now()) " "ON CONFLICT (id) DO UPDATE SET checked_at = now()" ) conn.commit() There is also a cheaper check that belongs in any code that opens a database connection and assumes it can write. Postgres can tell you whether the connection you hold is on a replica:
SELECT pg_is_in_recovery(); If that returns true, you are talking to a replica, and any write you attempt will fail. In Go or Python, run it once when a connection is opened for writes, and refuse to proceed if the answer is not what you expect. It costs one query. It catches exactly this class of mistake: the connection is healthy, the database is healthy, and the role is wrong.
The third guard is a habit, not code. When you test a failover, do not test that traffic moves. Test that a write lands. Fail over, submit a real update through the new path, read it back from the authoritative record, and only then call the test a pass. A balancer that answers pings is not a balancer that routes writes correctly. The RIPE incident is a case study in the difference.
What to take away
The failover worked. The standby took over, the traffic moved, and the system stayed up. Every part of the safety machinery did its job except the one line that said where the writes go. And because the old connections kept working, the fault hid behind partial success for eight hours.
If you run a primary and a replica, the question is not whether your failover works. It is whether you have ever watched a write travel the standby path and land. If the answer is no, you have the same untested configuration RIPE NCC had: a spare route that has never carried a real update, waiting for the day it has to.
When was the last time you failed over on purpose and proved a write landed on the other side?
Sources: RIPE NCC status page, "Intermittent RIPE Database update failures", postmortem posted 3 November 2026, incident window 28 October 2026, 11:42 to 19:40 UTC. The 30 October update on failed reverse-delegation requests is from the same incident record.
