Four Failure Domains - the cover: four nested layers, continent, provider, datacenter, zone

Four Failure Domains

Zone, datacenter, provider, continent: what a system feels at each level - latency, loss, a split, the whole level gone - and what the clouds themselves say about it.

👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.

This part is conceptual. The ordering of link kinds at the end is built; the failure-scenario runs across these levels are planned, not run.

This is part 2 of the series on testing cross-zone, cross-datacenter and cross-provider behaviour on a laptop. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.

One ladder

An availability zone, a datacenter, a provider and a continent sound like four separate topics. They are one ladder. Each step is a failure domain one size larger than the last, and each step up changes two things at once: how far apart the pieces are, and what can go wrong between them.

Four nested tiles: a continent holds a provider, which holds a datacenter, which holds a zone; each layer labelled with what it adds - distance, a new way to fail

The value of seeing it as a ladder is that one design can cover all four. Copies of a service are placed on the ladder; the network between them is described by the ladder; failures are cut at a step of the ladder. Part 1, One Login For Every Stand, gave every local environment one way in. This part is the map of where the copies behind that door can live, and what each place costs.

Zone

A zone is the smallest domain that a cloud promises to keep independent. AWS describes an Availability Zone as one or more discrete data centers with separate and redundant power, networking and connectivity, far enough apart to avoid correlated failures and close enough for synchronous replication with single-digit millisecond latency (AWS).

For a system, a zone means:

  • latency - near zero; nobody designs around it;
  • loss of a zone - the copies in it are gone; a quorum spread over three zones survives one.

The second line is the whole reason for three. Two copies of three is a majority; one of two is not.

An animation: three zone tiles each holding a node; one zone goes dark; the counter reads 2 of 3, quorum holds

Datacenter and metro

The next step is a whole site, or a metro of sites. Latency is small but no longer zero, and it is the first step where a timeout chosen on a laptop can be wrong in production. Synchronous replication is still possible inside a metro; it is simply slower.

A datacenter lost is a zone lost, only bigger: everything in it at once, including the things that were assumed to be "elsewhere" because they were in another zone of the same site.

Provider

A provider is a different kind of step. It is not further away so much as differently owned. Everything a provider runs - its API, its DNS, its network, its control plane - can fail together, and when it does, every site of that provider goes with it.

Two halves: on the left the same provider - one API, one DNS, one control plane, all sites fail together; on the right a second provider - separate failure, but a second integration to own

For a link between two places, the provider question has only two answers: the same provider or a different one. That is the whole axis.

Continent

The top step is the one physics decides. Between continents the round trip is many times larger than inside a region, and no configuration makes it smaller. Google's guidance says it plainly: communication within regions will always be cheaper and faster than communication across regions (Google Cloud).

At this step synchronous replication stops being a design choice. The main risk becomes the split: a cut between two continents leaves two halves that both believe they are the system.

A strip of three panels: a cut between two sides; two sides, each still running; the minority refuses writes, so there is never a second leader

What a system feels

Put the four steps side by side and the effects line up:

Level Latency Loss on the link A split The whole level gone
zone near zero rare unlikely copies in it lost; a quorum of three zones holds
datacenter / metro small rare possible everything in the site at once
provider depends on placement depends on placement possible every site of that provider, its API and DNS
continent many times larger more likely the main risk half the system on the far side

A matrix: four rows - zone, datacenter, provider, continent - against four columns - latency, loss, split, level gone; cells shaded from calm to vermilion as the level grows

The latency column reads as words, not numbers, on purpose. The order matters more than any single figure, and the order is the same in every cloud.

Four columns rising left to right: zone near zero, metro small, continent larger, inter-continent many times larger

What the clouds say

Three providers, one picture:

  • AWS - Regions are isolated and independent from other Regions; this separation limits service failures to a single Region, and resources do not exist in another Region unless replicated (AWS).
  • Google - zones are designed to minimize correlated failures of power, cooling and networking; the recommendation is to deploy across multiple zones and multiple regions (Google Cloud).
  • Azure - a zone is a logical group of physically separate datacenters, the target round trip between zones is under about two milliseconds, and zones do not protect against the loss of a whole region (Azure).

Three quote tiles side by side: AWS, regions isolated, failure limited to one region; Google, spread across zones and regions; Azure, zones do not protect against a region outage

None of them says "zones are enough". All three say: know which step of the ladder each copy is on.

A ladder a tool can read

For a test, the ladder has to become data. In our topology emulation, the kind of every link is derived from where its two ends are, in a fixed order:

intra-dc < intra-metro < intra-continent < inter-continent provider: same | cross (continent ranks above provider)

A flow of four tiles left to right: intra-dc, intra-metro, intra-continent, inter-continent; a pill under them reads provider: same or cross

Continent ranks above provider: two sites of different providers on one continent are closer than two sites of one provider on different continents. The provider is only ever "same" or "cross". That ordering is built; how it is used is part 3.

On a laptop

On a laptop each step of the ladder is a network: a subnet for a zone, a larger one for a site, a rule for every link between them. The failures become commands: take a zone down, isolate a site, cut a continent off. The runs of those scenarios are planned - the owner postponed them - so this part stops at the map.

What I do not claim

  • Not that any of these failures has been run across the levels. They are planned.
  • Not exact latencies for any level. The order is the point; the numbers belong to each cloud's documentation.
  • Not that a laptop network behaves like a cloud network. It is an emulation, and part 3 says how close it got.

What of this is method

  • One ladder, four steps. Zone, site, provider and continent are one design, not four.
  • Know the step of every copy. A quorum is only as good as where its members are.
  • Make the ladder data. A test tool can only break what it can read.

Cross-anything on a laptop - Part 2. Next: A Planet On A Laptop.

If you spread copies across zones, do you know which site each zone is in? If you run on two providers, what fails together that you thought would not? If you have lived through a continent split, which side kept writing? Which step of the ladder does your system assume will never fail?