What Paxos, VR, Zab, and Raft teach us. You may never implement consensus yourself. You still need its mental models to choose databases, coordination services, and infrastructure that fail the way you expect. At 2:13 a.m., your monitoring says the service is healthy. The API is answering. The database has replicas. Every dashboard is green enough to let you hope the pager was a mistake. But two nodes disagree about who the leader is. One accepted a write just before the network split. Another is about to accept a conflicting write. Both are alive. Both are behaving rationally with the information they have. And somewhere between them sits a question that a health check cannot answer: When several machines remember different versions of reality, which one becomes the truth? That is the problem consensus algorithms solve. Most developers will never implement one—and probably should not. Yet many of us choose and operate systems built on them: databases, configuration stores, service-discovery platforms, schedulers, control planes, and distributed locks. If you understand the ideas underneath those tools, their tradeoffs stop looking like mysterious product limitations. They become predictable consequences of design. This article looks at four protocols: Multi-Paxos, Viewstamped Replication, Zab, and Raft. The goal is not to reproduce their proofs or prepare you to write one over a weekend. It is to learn four ways of thinking about the same hard problem—and then use those mental models when evaluating real infrastructure. Who this is for This is for backend developers, platform engineers, SREs, architects, and technical leads who choose or operate distributed systems but do not build consensus protocols themselves. You should leave with enough intuition to: Ask better questions during a database or coordination-tool evaluation. Understand why a system may reject traffic while several machines are still running. Reason about leader changes, stale reads, write latency, quorum size, and recovery time. Recognize which complexity a tool has absorbed—and which complexity it has handed back to you. Suppose three replicas maintain the same state. A client sends SET plan = pro. The replicas must agree not only that the command happened, but where it belongs in the history. If command number 42 is SET plan = pro on one replica, command number 42 cannot be DELETE account on another. The usual mechanism is a replicated log. Each entry is an ordered command. Once a quorum—typically a majority—has accepted an entry under the protocol's rules, the cluster can treat it as committed and apply it to a state machine. A majority matters because any two majorities overlap. In a five-node cluster, a quorum of three can tolerate two crashed or unreachable nodes. Two separate groups cannot both form a majority, so a network partition cannot safely produce two independent committed histories. That safety has a price. If no majority can communicate, a correctly designed cluster may stop accepting writes. The machines are up; the service is unavailable by choice. This is not necessarily a bug. It may be the system refusing to invent two truths. One more boundary: the four protocols here are designed primarily for crash faults—nodes fail, restart, or become unreachable—not Byzantine behavior where a participant lies or sends maliciously contradictory messages. Why exactly these four? These four are useful together because they form a compact tour through the major design ideas behind crash-tolerant replicated state machines. Protocol The design question it highlights The lesson worth keeping Multi-Paxos How can repeated decisions reuse stable leadership? Separate safety from the optimizations that make a protocol fast. Viewstamped Replication (VR) How does a primary-backup system change leaders without losing committed work? Recovery and view change are part of the protocol, not cleanup after it. Zab How do we preserve a strict prefix of history for a coordination service? A workload-specific ordering contract can shape the entire protocol. Raft Can the same safety goals be expressed as understandable, enforceable rules? Comprehensibility is an operational feature. They are not the only important consensus algorithms, nor four entries in a leaderboard. They overlap substantially. The value of comparing them is seeing where each one places structure: in ballots, views, epochs, terms, logs, leaders, and recovery rules. Multi-Paxos Basic Paxos reaches agreement on one value. Roughly, a proposer first asks acceptors to promise not to accept older proposals, then asks them to accept a value under a numbered ballot. The subtle rules ensure that once a value can be chosen, a later ballot cannot replace it with a conflicting value. Doing that full exchange for every log position would be expensive. Multi-Paxos adds the practical move: establish a stable leader for a ballot, then let that leader drive many log positions without repeating the prepare phase every time. In steady state, the shape becomes: A leader receives a command. It proposes the command for the next log position. A quorum of acceptors accepts it. The value becomes chosen and can be learned or applied. The important insight is not “Paxos is complicated.” It is that the protocol's safety core is more general than the leader-based system we usually run. Stable leadership is an optimization for progress and efficiency. When leadership becomes unstable, the system falls back into the harder work of establishing a newer ballot and discovering what may already have been chosen. Where this mental model helps Multi-Paxos is a good lens for highly optimized or custom replicated services. Implementations can pipeline, batch, use flexible quorum arrangements, and make other choices around the safety core. That flexibility is powerful, but it increases the amount of protocol detail an implementation team must get right. When a vendor says its system is “Paxos-based,” the label is only the beginning. Ask what kind of Paxos, how leadership works, how log gaps are repaired, how membership changes are handled, whether reads use quorum or lease mechanisms, and what operators can observe during recovery. What to remember: A general safety foundation can support many optimized implementations, so two “Paxos-based” products may behave very differently in production. Viewstamped Replication (VR) Viewstamped Replication starts from a primary-backup picture. At any moment, the replicas operate in a numbered view, and one replica is the primary for that view. During normal operation: The client sends a request to the primary. The primary gives it the next operation number and sends a prepare message to backups. Once enough replicas acknowledge it, the operation commits. Replicas execute committed operations in order, and the primary replies to the client. The interesting part begins when the primary appears to fail. Replicas move to a higher view, exchange information about their logs, and the new primary constructs a state that preserves committed operations. In the revisited protocol, the primary for a view is selected deterministically from the view number and group membership. VR makes an architectural point that is easy to miss: a leader change is a state-transfer protocol. Electing a name is not enough. The new primary must know which history it is allowed to continue. Where this mental model helps VR is especially useful for understanding primary-backup storage and replicated services. Even when a product does not literally implement VR, its vocabulary—views, operation numbers, commit numbers, normal processing, view change, recovery—gives you a clean way to interrogate failover. Ask: What state is transferred before a promoted replica serves writes? Can an out-of-date replica become primary? When does a client retry become a duplicate operation? How does the system distinguish a recovering replica from a participant in the current view? What to remember: Failover is safe only when leadership and history move together. Zab Zab—ZooKeeper Atomic Broadcast—was built for Apache ZooKeeper's primary-backup architecture. ZooKeeper is not merely storing independent keys. It provides a coordination namespace where the order of changes matters: create a membership node, update configuration, delete a lock contender, trigger watchers. Zab therefore emphasizes a totally ordered stream of state changes and divides the protocol into two broad modes: Recovery, where a leader is established and replicas synchronize on a valid history. Broadcast, where the leader proposes transactions, followers acknowledge them, and committed transactions are delivered in order. Transactions carry a zxid, a monotonically ordered identifier containing an epoch-related component and a counter. A prospective leader cannot simply start appending after an election. It must complete the synchronization work that gives the ensemble a safe common prefix. That is Zab's central lesson: sometimes the product's data model tells you which consensus property deserves the spotlight. ZooKeeper's hierarchical namespace and coordination semantics make ordered broadcast—not isolated agreement on unrelated values—the natural abstraction. Where this mental model helps You encounter Zab when using ZooKeeper directly or operating platforms that rely on it for metadata and coordination. The practical questions are not “Is Zab better than Raft?” but “Do ZooKeeper's semantics fit this job?” ZooKeeper is well suited to small coordination data, configuration, membership, and synchronization primitives. It is not a general replacement for a high-volume application database. You should also examine read semantics separately from write ordering: a protocol can totally order updates while a product still offers local reads with freshness tradeoffs. What to remember: Choose a system whose ordering contract matches the meaning of your data, not one whose algorithm name sounds strongest. Raft Raft was designed around understandability. It decomposes replicated-log consensus into leader election, log replication, and safety, then makes the leader-follower relationship deliberately asymmetric. In normal operation: A client sends a command to the leader. The leader appends it to its log and sends AppendEntries RPCs to followers. After the entry is safely replicated according to Raft's commit rule, the leader applies it and replies. Followers learn the commit position and apply the same entries in order. Terms act as logical eras of leadership. Each log entry records the term in which it was created. During an election, a voter rejects a candidate whose log is less up to date than its own. Once elected, the leader uses the previous log index and term in AppendEntries to detect divergence, then repairs followers by replacing conflicting uncommitted suffixes. The nuance matters: a new leader does not win because it has every byte any replica has ever seen. It wins under rules designed to ensure that committed entries cannot be lost. Uncommitted entries may disappear, which is why a client timeout does not always tell you whether an operation committed. Raft's greatest contribution may be sociotechnical: a protocol that engineers can explain, review, test, and debug is less likely to be implemented incorrectly. Understandability is not cosmetic when the code decides which data survives a failure. Where this mental model helps Raft appears under widely used infrastructure such as etcd and Consul; CockroachDB uses many Raft groups to replicate ranges of data. Knowing the shared foundation helps, but it does not make these products interchangeable. Their data models, transaction layers, read paths, placement, snapshots, reconfiguration, and operational tooling differ enormously. What to remember: An understandable consensus core reduces one category of risk, but the surrounding distributed system still determines the user experience. The four protocols in one operational picture Lens Multi-Paxos Viewstamped Replication Zab Raft Leadership era Ballot View Epoch Term Normal-path coordinator Stable proposer/leader in practical deployments Primary Leader Leader Replicated object Sequence of consensus instances, commonly used as a log Ordered operation log Totally ordered transaction stream Ordered log Recovery emphasis Discover and preserve already chosen values Build a safe new view from replica state Synchronize a safe history before broadcast Elect an up-to-date candidate; repair follower suffixes Design personality General and optimization-friendly Primary-backup made explicit Coordination-workload-specific Structured for understandability This table is a map, not a benchmark. Performance depends on implementation, batching, storage, network topology, durability settings, read mode, workload, and cluster health. “Raft versus Paxos” is rarely a useful procurement question by itself. How to apply this knowledge without implementing anything Imagine you are choosing a distributed database for an order service. Product A advertises Raft. Product B advertises Paxos. The tempting conclusion is that the protocol name settles the decision. It does not. Use the algorithms to generate better questions. 1. What is the unit of consensus? Is there one log for the whole cluster, one group per shard, one group per data range, or a metadata consensus group plus separate data replication? This determines where contention appears and how failure domains interact. 2. Which operations actually pass through consensus? Writes probably do. What about reads? Are they linearizable, lease-based, quorum-based, or served locally and potentially stale? Does a transaction spanning several consensus groups require another coordination layer? 3. What happens without a quorum? Will the system stop writes, serve stale reads, fail over elsewhere, or expose a tunable consistency mode? A safe refusal can be preferable to silent divergence, but your product must be designed for that refusal. 4. What does “acknowledged” mean? Was the operation accepted by memory, written to durable storage, replicated to a majority, committed, or applied to the state machine? Those are different milestones. Ask what survives a leader crash immediately after the client receives success. 5. How is leadership moved? Look for election timeouts, planned leadership transfer, fencing, catch-up requirements, and behavior under asymmetric packet loss. Failover time is often a distribution, not a single number. 6. How does a slow replica recover? Does it replay a log, install a snapshot, fetch a checkpoint, or rebuild from another store? Can recovery saturate the same network and disks serving production traffic? 7. How does membership change? Adding and removing voters is consensus about who participates in consensus. Safe reconfiguration is not equivalent to editing a host list. Understand the product's supported procedure. 8. Can operators see protocol state? You want metrics for leader changes, term/view/epoch changes, commit lag, proposal latency, quorum health, snapshot transfer, and rejected requests. A correct protocol hidden behind poor observability can still create a terrible incident. What these algorithms do not solve for you Consensus is a foundation, not a complete distributed system. It does not automatically give you: Multi-key or cross-shard transactions. Exactly-once side effects. A correct retry strategy for ambiguous client timeouts. Good geographic latency. Elastic scaling. Protection against malicious replicas. Correct application invariants. Painless upgrades and disaster recovery. For example, if a payment request times out during a leader change, consensus may preserve the command perfectly while the client cannot tell whether it committed. The application still needs an idempotency key and a safe retry contract. Likewise, consensus can order reserve item and charge card without guaranteeing that the business workflow across two external systems is atomic. That is an application-level problem. A small real-life decision exercise Suppose you need three capabilities: Service discovery and a small amount of strongly consistent configuration. A globally scaled transactional database. A custom replicated control plane. The protocol perspective changes how you evaluate them: For coordination, you inspect ZooKeeper, etcd, or Consul semantics: watch behavior, session or lease models, read consistency, quorum loss, and operational maturity. Zab versus Raft is context, not the final score. For the database, you ask how many consensus groups exist, how transactions cross them, where leaders are placed, and what geographic topology does to commit latency. For the custom control plane, you should strongly prefer a mature library or service over implementing a paper. If custom behavior is truly necessary, Multi-Paxos shows the optimization freedom available; Raft shows the value of constrained, reviewable state transitions. In all three cases, knowing consensus helps you see the system you are actually buying. The deeper lesson: failure behavior is part of the API We often evaluate infrastructure by its happy-path interface: SQL syntax, key-value operations, SDK quality, or throughput in a benchmark. Distributed systems reveal their real contract when messages are delayed, leaders restart, disks stall, and clients retry. Multi-Paxos teaches us to distinguish a safety core from its performance optimizations. Viewstamped Replication teaches us that a new leader must inherit a safe history, not merely a title. Zab teaches us that ordering should match the workload's meaning. Raft teaches us that understandability can improve implementation and operations. You do not need to implement any of them to benefit. You need to recognize the questions they force every distributed tool to answer: The questions worth asking: Who may speak for the cluster? Which history survives? What can progress without a majority? And how will we know what happened after the network heals? The next time a product page says “strongly consistent” or “powered by Raft,” do not stop at the label. Ask for the failure story. Because at 2:13 a.m., the failure story is the product. Further reading Leslie Lamport, Paxos Made Simple Barbara Liskov and James Cowling, Viewstamped Replication Revisited Flavio P. Junqueira, Benjamin C. Reed, and Marco Serafini, Zab: High-performance broadcast for primary-backup systems Diego Ongaro and John Ousterhout, In Search of an Understandable Consensus Algorithm etcd FAQ: Raft and cluster behavior Consul documentation: Consensus protocol