The Committee Pretending to Be One Machine

01The Week the Training Wheels Come Off

Look back at what you’ve rented so far, and notice a pattern. The bucket in Notes 04 promised your photo would survive almost anything — eleven nines — because copies of it live in three buildings. Notes 08’s manager promised that a dead pod becomes a boring log line, because a loop keeps comparing a wish list to reality. Last week’s landlord promised failover: when the database machine dies at 2 a.m., a standby copy is promoted in about a minute, and you read about it over coffee. Every one of those promises has the same fine print, and until today we have let someone else read it for you.

Here is the fine print, in one sentence: there is no single giant machine behind any of these services — there is a committee of ordinary machines agreeing, over an unreliable network, to pretend to be one. The pretense is the product. And every hard thing in this note — copies that briefly disagree, data split across machines, a famous trade with no third option, voting rules that sound like a town council — is a problem the committee must solve to keep up the act.

Why open the hood at all, if the landlord runs the committee for you? Three reasons, and you have personally met all three. Your Project 2 datastore keeps its promises — or quietly doesn’t — because of what’s in this note. Notes 04’s stale read, the group-chat photo your friend couldn’t see yet, was this. And last week’s read replicas, the checkbox that adds “extra copies that answer read-only questions,” is this too — and after today you’ll know exactly what that checkbox can and cannot promise. You will almost never build what’s in this note. You will choose among its consequences constantly.

02Replication — the Same Data, on Several Machines

Replication is the committee’s first act: keep the same data on more than one machine. It does exactly two jobs, and you’ve already paid for both. Job one: survive a machine dying. That is Notes 04’s three buildings — a copy you can lose is a disaster you don’t have. Job two: serve more readers. That is last week’s read replicas — ten thousand people can read the menu at once if ten machines each hold a copy.

Reads love copies. The catch — the entire catch — is writes. The moment the data changes, every copy must change too, and they cannot all change in the same instant, because the news travels over a network. The most common arrangement is the simplest honest one, and your landlord’s failover and read replicas are both built on it: leader and followers. One machine — the leader — is the only door writes may enter. The followers watch the leader and copy each change, a beat later. Readers may be served by any copy.

That “a beat later” has a name: replication lag — usually milliseconds, sometimes seconds when the system is busy. And here is a debt paid: the group-chat photo from Notes 04, the one on your screen that your friend across town couldn’t see yet, now has a mechanism instead of a shrug. Your write landed on the leader. Her read was answered by a follower that hadn’t copied it yet. Two machines, both telling the truth as they know it, briefly knowing different truths. That is all a stale read ever was.

2.1 When the leader dies

Followers are easy to lose — they hold nothing unique. But the leader is the one door for writes, so when it dies, the committee has a real moment. For a few seconds nobody is taking writes at all: that’s the failover gap, and it is the “about a minute” inside last week’s landlord promise. Then the most caught-up follower is promoted — handed the crown, declared the new door. If the dead leader had accepted writes the followers hadn’t copied yet, those writes are simply gone. Lag isn’t just staleness; during a failover, lag is the size of what you can lose.

And one scenario is scarier than a dead leader: a leader that isn’t actually dead. Suppose the old leader didn’t crash — it was only cut off for a minute, and it comes back still believing it wears the crown, while a follower has already been promoted. Now two machines both think they’re the leader, both willing to accept writes, and the data forks into two histories. The name for this is split brain, it is among the worst things that can happen to a data system, and preventing it is precisely what §05’s voting is for. Watch it happen safely below.

Four experiments worth your five minutes. One: set the lag high, post a photo, and immediately read from a follower — a stale read, on demand; wait and read again — fresh. Two: post two photos and kill the leader before the copies land — count what the promotion loses. Three: kill the leader having written nothing — promotion still happens, and nothing is lost, because there was nothing to lose. Four, the main event: after a promotion, wake the old leader up and watch the committee briefly have two heads. Real systems survive that moment only because of §05.

03Partitioning — When the Data Itself Won’t Fit

Replication copies everything onto every machine, so it never solves one problem: what if everything doesn’t fit? A billion user profiles will not fit on one machine, and copying them onto five machines gives you five machines that are each too small. The answer is the other great move: split the data. Each machine holds a piece, and the pieces together make the whole. The word is partitioning — you’ll also hear sharding, same idea. (Last week’s wall two, “the one-machine ceiling,” is exactly what this breaks through — this is how the NoSQL stores spread across fifty machines.)

There are two honest ways to split, and both are things you’ve done with physical stuff.

Split by range — the filing cabinet. Drawer one holds customers A–F, drawer two G–M, and so on. The lovely property: neighbors stay together, so “every customer from A to C” opens one drawer. The ugly property: drawers run hot. If one drawer holds the data everyone wants today, that one machine melts while the others nap.

Split by hash — deal the cards. Run each record’s key through a blender that assigns it a pile, effectively at random, and deal the piles around the table. The lovely property: the piles stay beautifully even. The ugly property: neighbors scatter — “every customer from A to C” now touches every pile, because the blender ignored the alphabet on purpose.

Two honest footnotes before we move on. First, the hot spot survives even the blender: hashing spreads users evenly, but if one single user is a celebrity whose profile the whole internet reads today, their key lands in one pile and that pile burns anyway — one famous key is a problem no splitting scheme fixes by itself. Second, resharding: when you must change how the data is split — more machines, different ranges — you are moving the filing cabinet’s contents between drawers while people keep asking for the files, and that single sentence is why teams dread it and why managed stores that reshard for you earn their premium.

Notice, finally, that real systems do both at once: the data is split into pieces, and each piece is replicated onto several machines. Pieces for size, copies for safety. Which means every piece has its own little leader, its own lag, its own failover — the committee, fractally, all the way down.

04CAP — the Trade the Network Forces

Now the famous one. You’ve seen that the committee’s machines constantly talk to each other — copying writes, confirming balances, comparing notes. Here is the fact the whole section rests on: the network between them will break. Not might — will. A cable gets cut, a router dies, a building loses its uplink for ninety seconds at 2 a.m. When committee members can’t reach each other, that’s called a partition (an awkward reuse of §03’s word — here it means the network tore, splitting the committee into groups that can’t talk).

The moment the network is torn, every machine still answering customers faces one question with exactly two answers. A customer asks it something. It cannot check with the rest of the committee. It can either refuse to answer until it can be sure — stay consistent, at the price of being unavailable — or answer from what it has — stay available, at the risk of serving stale or even conflicting data. That choice is the whole of the CAP trade. Consistency or Availability, when the Partition comes.

The classic story makes it physical. A bank has two branches, uptown and downtown, sharing one account that holds its last $100. The phone line between the branches is cut. A customer walks into each branch and asks to withdraw the $100. Each teller faces the choice alone: refuse — “I can’t confirm your balance right now, try again later” — and the bank is consistent but closed for business. Or pay — each teller trusts the balance they last knew — and the bank just handed out $200 of a $100 account. That is CAP, entire. Everything else written about it is commentary.

Run both endings. Cut the line under refuse when unsure and try to withdraw: the teller says no — not because the money isn’t there, but because she can’t be sure it is. That refusal is why your card sometimes says “try again later”: a bank would rather be briefly closed than briefly wrong. Then reset, choose answer anyway, cut the line, and withdraw at both branches: $200 leaves a $100 account, and when the line is repaired the branches compare notes and find the fork. Someone writes an apologetic letter. Neither teller made a mistake — each followed her policy perfectly. The policies just price wrongness differently.

Now connect it backwards. Notes 04’s eventual consistency — “all copies agree, shortly” — is choosing availability: the bucket answers from the copy it has, even if that copy is a beat behind, and the copies converge when they can. A money-moving database that refuses writes when its committee can’t agree is choosing consistency. Neither is the “right” choice. A stale photo caption costs nothing; a double-spent $100 costs exactly $100. The only wrong move is not knowing which choice your system made.

05Consensus — How a Committee With No Boss Decides

One problem left, and it’s the deepest one. Replication assumed a leader. But who decides who the leader is — especially at the worst moment, right after the old leader died? There is no boss machine to appoint one; a boss machine would just be a leader we’d need to replace when it dies. Somehow a committee of equals, talking over a network that drops messages, must agree on one answer — “Machine 3 is now the leader,” or “write #42 comes before write #43” — and never un-agree. That problem is called consensus.

The solution is the one humans landed on for the same problem: majority voting. The rule has one line: nothing counts as decided until more than half the committee has written it down. That’s it. And the single line buys a remarkable guarantee, which you can check by drawing: take five members; any group of three overlaps any other group of three — there aren’t enough members to build two separate majorities. So if “Machine 3 is leader” was written down by a majority, no other majority can exist to write down a different leader. Someone in the overlap would have to sign both, and they won’t. The committee can never decide two different things. That overlap — one picture, no proof needed — is what keeps Plate 2.2’s split brain from happening in real systems: a machine may act as leader only while a majority currently backs it, so the woken-up old leader finds it cannot gather one and stands down.

The rule also explains a number you’ll see on every vendor page: committees come in 3s and 5s, never 4s. More than half of 4 is 3 — so a four-member committee tolerates only one dead member, exactly like a three-member one. The fourth machine costs money and buys nothing. (The word for “more than half, gathered and agreeing” is a quorum — now you’ve been introduced; we’ll mostly keep saying “majority.”)

One honest caution about how good this can ever get. Two generals on two hills must attack together or not at all, and can only communicate by messengers who are sometimes captured. The first general sends “attack at dawn” — but now he needs to know the message arrived, so he needs a confirmation back, and the second general needs to know that arrived, forever. Perfect certainty over an unreliable line is impossible — provably, not just annoyingly. So real systems don’t chase it; they settle for majority votes plus patient retries, which turn “impossible to be certain” into “almost always fine, shortly.” The named recipes for this voting dance — who calls elections, how ties break, how the log stays in order — are Raft and Paxos, and knowing those two names plus everything above is genuinely enough for this course.

06What You Actually Do With All This

You will not implement Raft, and nobody will ask you to. Here is the practitioner’s whole cheat sheet — what this week’s ideas look like when they reach your keyboard:

• Rent the committee. The landlord already runs the voting, the failover, the resharding. The managed database, the managed cache, the managed control plane — their premium is largely the salary of this entire note. Same rule as Weeks 8 and 9: pay a premium, skip a pager.
• Know which knob you’re turning. When a database setting says consistency level — “strong” vs. “eventual,” “read from leader” vs. “read from any copy” — that is the C-versus-A dial from §04, wearing a dropdown. Turn it knowing which reads in your app may lie and for how long.
• Keep writes on one leader unless you truly can’t. One door for writes is the arrangement whose failures you can reason about. Multi-leader setups exist and have their place; they also invite the data to fork, and “truly can’t” is rarer than it sounds.
• Three replicas, odd committees. Three copies of data (Notes 04 said so for files; it holds for databases), and 3 or 5 voters for anything that elects a leader — never 4, for the reason you can now give in one sentence.
• Design for the 90-second break. The network between your services will tear for a minute and a half at 2 a.m. For each service, you should be able to answer: during those 90 seconds, does it refuse, serve stale, or queue up work and wait? “I never thought about it” is the only wrong answer.

And the honest closer: this note is the map, not the territory. People build entire careers inside §05 alone. What you now own is the thing the map is for — when a system behaves strangely, you can say which of these four problems you are looking at, and that is most of what “knows distributed systems” means outside a PhD defense.

07Self-Check

Six quick questions. Pick an answer to see whether you’re right and why. Nothing is recorded; this is for you.

08Before Session 19 — November 9

Three things, none longer than a coffee. The list remembers your progress on this device.

• Drive Plate 2.2 until you can produce, on demand: a stale read, a write refused during the failover gap, and at least one lost write — and say in one sentence what the lag slider had to do with the loss. Expect to drive it in Session 19’s opening round.
• Drive Plate 4.2 to both endings — the refusal and the double-spend — then answer for your own Project 2 datastore: during a network break, does it refuse or answer anyway? (Your provider’s docs say; the phrase to search is “consistency.”)
• Keep Project 3 moving. You already built everything it asks for once by hand — this is the week it starts feeling less like homework and more like a superpower.

09Key Terms

Replication

The same data kept on several machines. Two jobs: survive a dead machine, serve more readers. The catch is always writes.

Leader and followers

One machine is the only door for writes; the others copy it, a beat behind, and may serve reads.

Replication lag

The beat between the leader and a copy — Notes 04’s group-chat photo, measured in milliseconds.

Stale read

An answer from a copy that hasn’t heard the news yet. True when written, old when read.

Failover / promotion

The leader dies; after a short gap with no door for writes, the most caught-up follower takes the crown.

Split brain

Two machines both believing they’re leader, forking the data. Prevented by majority voting — there can’t be two majorities.

Partitioning (sharding)

Splitting the data itself across machines when it won’t fit on one. Pieces for size, copies for safety.

Range split

The filing cabinet: drawers A–F, G–M… Neighbors stay together; drawers can run hot.

Hash split

Deal the cards: a blender assigns each key a pile. Even piles; range questions touch every pile.

Hot spot

One drawer, pile, or single famous key taking all the traffic while the rest nap.

Resharding

Changing how the data is split — moving the files while people keep asking for them. Dreaded, accordingly.

Network partition

The network tears and committee members can’t reach each other. Not “might happen” — will.

CAP

During a partition, each machine must choose: refuse until sure (consistent, unavailable) or answer from what it has (available, maybe wrong).

Eventual consistency

Choosing availability: copies may briefly disagree, then converge. Notes 04’s promise, now with its mechanism.

Consensus

A committee of equals, over an unreliable network, agreeing on one answer and never un-agreeing.

Quorum (majority)

More than half the committee. Any two quorums overlap — the one-picture reason nothing gets decided twice.

Raft / Paxos

The named recipes for the voting dance: elections, tie-breaks, keeping the write log in one order.

Two Generals

Why perfect certainty over an unreliable line is impossible — so real systems settle for majorities plus retries.

etcd

The small majority-voting committee under Kubernetes’ control plane — Notes 08’s wish list lives in one.

No term matches that filter.