Leave a Note, Don’t Call

01The Problem With Phone Calls

Think about how your services have talked to each other so far. The web pod calls the API pod. Checkout asks the database. Every one of those conversations has the same shape: A asks B a question, and then A waits on the line until B answers. We never gave that shape a name because there was nothing to contrast it with. It has one — a synchronous call — but “phone call” says it better. You dial, you ask, and you stand there holding the phone.

Phone calls fail in three flavors, and you have met all three this term. B is down — Notes 08’s crash loop: the callee dies on every start, and until someone fixes the bug, every caller gets a dial tone. B is slow — a call is exactly as slow as the slowest thing on the line, so the caller’s customer stares at a spinner while B thinks. And B is drowning — Notes 03’s traffic spike: callers pile up faster than B can answer, and the autoscaler that could help is minutes away (boot time is scaling latency) while the callers are seconds impatient.

But the deeper trap is quieter than any of those. When A calls B and waits, A has chained its fate to B’s. Picture checkout: charge the card, update inventory, email a receipt. Done as three phone calls, the purchase succeeds only if all three answer. So the day the email service hiccups for thirty seconds, every purchase in those thirty seconds fails — card fine, inventory fine, sale dead. The receipt is the least important step in the chain, and it just killed the sale. That is what waiting on the line costs: every callee’s bad day becomes yours, instantly.

02The Queue — the Note on the Door

Here is the whole move, and it is smaller than the problem deserves: instead of calling B, A writes the ask down and leaves. “Send a receipt to sam@… for order 741” goes onto a slip of paper — a message — and the slip goes into a queue: a waiting line of notes, oldest at the front. Checkout drops the note and goes back to work. The email service pulls notes off the front at whatever pace it can manage. Nobody holds a phone. Last week ended by promising you committees that leave notes for each other — this is that, and the rest of this week is the fine print on the notes.

What does the note buy you? Three things, and each one is a failure flavor from §01, retired.

The spike flattens. A lunch rush drops five hundred notes in a minute on a worker that does sixty a minute — and nothing breaks. The line grows, the worker chews through it steadily, and ten minutes later the line is empty. Nobody was refused, nobody saw an error; some receipts were a few minutes late. The queue is a shock absorber: spiky arrivals in, steady work out, the line in the middle soaking up the difference. (In Notes 03 the same spike meant drowned callers until the autoscaler woke up.)

Downtime stops being loss. If the email worker is down for ten minutes — crashed, redeploying, stuck in a crash loop someone is calmly fixing — the notes do not fail. They wait. The door doesn’t have to be answered for the note to be safely on it. The worker comes back, finds the backlog, drains it oldest-first, and from outside the system merely ran a little late.

Fates come unchained. Checkout no longer succeeds or fails with the email service. A late receipt is now exactly what it should always have been: a late receipt — not a dead purchase. Each service gets to have its bad day alone.

Every provider rents you this exact thing: SQS on AWS, Pub/Sub on Google, Service Bus on Azure — same note, same door. One more name you’ll meet in job postings: Kafka, the heavy-duty flavor — less a waiting line than a log everyone reads at their own bookmark — and for this course, that one sentence is enough.

Three experiments worth your five minutes. One: send a rush and just watch — the amber spike towers over the steady blue line, and the gap piles into the waiting hill, which drains at exactly the worker’s pace. Two: send a rush, kill the worker mid-drain, count to five, revive it — the hill freezes, nothing is lost, the drain resumes. Three: set the worker slow and send two rushes back to back — the line gets long, and long is the honest cost of a queue: nothing fails, things get late. Whether late is fine is a product question, not an infrastructure one.

03Two Shapes of Note-Leaving — Tickets and the PA System

Notes get left in two shapes, and you have stood inside both.

The first is the work queue, and it is a diner at noon. Order tickets pile onto the rail; whichever cook is free pulls the next one. The rule that makes it a work queue: each ticket is cooked by exactly one cook. Two cooks don’t both make table nine’s burger — they split the rail. Which gives you scaling for free: line too long? Hire another cook. Same queue, more pullers, faster drain. That’s §02’s receipt worker, multiplied.

The second is the broadcast — also called a topic, or publish/subscribe — and it is the school PA system. One announcement, and every subscriber hears their own copy. “Order 741 placed” rings out once; the receipts team does their thing, the inventory team theirs, and the analytics team quietly writes it down. Three jobs from one announcement, none aware the others exist.

That second shape has a grand name you should recognize on sight: event-driven architecture. In plain words: instead of the checkout service phoning four departments with four instructions, it announces one fact — “order 741 happened” — and departments subscribe to the facts they care about. Checkout stops knowing who its listeners are at all. It tells the truth into the PA and goes back to work.

What this buys is real: next quarter, when a fraud-detection team wants to watch orders too, they subscribe — a fifth department gets added without touching checkout, without redeploying it, without checkout ever learning fraud exists. What it costs is just as real, so say it honestly: once facts fan out to subscribers who fan out further, nobody can see the whole flow in one place anymore, and “where did my order go?” becomes detective work across four services’ logs. The tooling that makes that bearable is Week 13’s whole topic — observability.

04The Fine Print — “Exactly Once” Is Not for Sale

Now the lesson of Notes 10 comes home to roost. The queue hands a note to a worker, then needs one more thing before it can throw its copy away: the worker must report back “done” — the acknowledgment, the ack. And that report travels over the same unreliable network as everything else in this course. You can already feel what’s coming.

Here is the honest story, four sentences long. One day the worker takes the note, finishes the job — the receipt actually sends — then crashes before the ack leaves. The queue hears silence, and silence after the work and silence instead of the work sound identical from its side of the wall. So the queue must assume the worst and hand the note out again. The work gets done twice.

Could the queue just… check more carefully? No — and you already know why. Being certain a message arrived, over a line that drops messages, is the two-generals problem from Notes 10 wearing an apron. It is not hard; it is impossible, provably. So every real queue makes the only promise physics allows: at least once. Not exactly once — at least once. Your note will occasionally arrive twice, on the worst possible day, and pretending otherwise is how systems double-charge real people’s cards. The rest of this note is what to do about it.

05Idempotency — Work That Survives Being Done Again

Here is the week’s hero word, worth learning to spell because it will be in your Project 4 review: idempotency. An action is idempotent if doing it twice leaves the world exactly as doing it once. That’s the entire definition. A third time? Same world. A hundredth? Same world.

You already live surrounded by idempotent actions. Pressing an elevator button that’s already lit — the elevator doesn’t come twice as fast. Setting the thermostat to 68 — do it five times, the room is 68. Now compare its evil twin, “raise the temperature by 3 degrees” — do that five times and the room is a sauna. Same intent, opposite safety. Set to a value survives repetition; add an amount compounds it. Or: highlighting a sentence that’s already highlighted. Still yellow. The repeat was harmless because the action describes a destination, not a step.

So how do you make real work idempotent? Two moves, and most systems use both. Move one: the done-list. Give every message an ID — “this is the charge for order 741,” not “charge $20” — and have the worker keep a list of IDs it has finished. Before doing anything, check the list. You’ve met this worker: she is Notes 09’s barista, glancing at her memory before walking to the kitchen — “order 741? already made it.” The duplicate gets a shrug and an ack, and nothing happens twice. Move two: phrase the work as SET, not ADD. Not “append a row,” but “make row 741 exist.” Not “raise by 3,” but “set to 68.” Work phrased as a destination can be repeated by accident, forever, safely.

Walk the money example both ways, because this is where it stops being philosophy. The note says “charge $20” and arrives twice: the card is charged $40, a customer is on the phone, and someone writes Notes 10’s apologetic letter. The note says “make order 741’s charge exist — and a charge keyed to 741 can only exist once” and arrives twice: the second delivery finds the charge already there, does nothing, acks, and nobody ever knows. Same network, same crash, same redelivery — all that changed is that the work was built to survive being done again.

Run the disaster first, then the fix. One: keys off, crash chance around 50%, place three orders — sooner or later a worker charges the card and dies before the ack, the queue re-delivers, and a duplicate $20 appears on the statement in red. Nobody misbehaved: the queue kept its at-least-once promise, the worker did what the note said. The work was fragile. Two: reset, turn the keys on, crank the crash chance to 100% — every single delivery crashes — and place an order. Watch the statement: exactly one $20, no matter how many times the note comes back, because every delivery after the first finds the charge already existing and just acks. The note eventually lands in the needs-a-human tray (that’s §06), but the customer was charged once.

06Retries — the Polite Way to Insist

So deliveries fail, workers crash, and notes come back. The response is obvious — try again — and the whole craft is in the adverbs: not instantly, and not forever.

Not instantly: back off. Wait one second before the first retry. Two before the next. Then four, then eight. A service that just failed is usually a service having a hard moment — give it air instead of trampling it the instant it stumbles. You have seen this rhythm before: Notes 08’s manager restarting a crashing pod with doubling waits. The rule there was never restart harder; the rule here is the same rule in new clothes: never retry harder.

And one word about rhythm: jitter. If a thousand callers all fail at the same moment and all retry exactly one second later, the retry is a stampede — so each waits a slightly random amount, and the herd arrives as a drizzle instead of a wave.

Not forever: the dead-letter queue. After some number of honest attempts — four is common — the note stops being retried and moves to a separate tray a human checks: the dead-letter queue. Because Notes 08’s crash-loop lesson applies to notes too: retries repair accidents, not mistakes. A note that failed once at 2 a.m. hit the 90-second network break — retry it and it sails through. A note that has failed ten times isn’t unlucky; it’s wrong — a malformed address, a bug, a card that doesn’t exist — and the eleventh retry fixes nothing. Accidents get retries. Mistakes get a human.

07Putting It Together — the Shop, Rewired

Here is the whole week assembled into one small architecture — the course’s running shop, rewired from phone calls to notes. Checkout takes the order and does the one money-critical thing itself: it makes order 741’s charge exist, keyed to the order ID in the landlord’s database, so even checkout’s own retries can’t charge twice. Then it announces one fact to a topic — “order 741 happened” — and answers the customer. Done in under a second, and it phoned nobody.

The receipt worker and the inventory worker each subscribe with their own queue, so a lunch rush piles up as backlog, not as errors. Every note carries the order ID, and every worker checks its done-list before working, so the redelivery §04 guarantees will eventually happen lands as a shrug and an ack. When the email provider has a bad afternoon, receipt notes retry at one, two, four seconds — with a little jitter so the herd doesn’t stampede — and after the fourth strike a note drops into the dead-letter tray, where a human finds it over morning coffee. And when the fraud team shows up in the spring, they subscribe to the same topic — checkout is never redeployed, never even told.

And the honest closer: you are not required to build this — this term. But Project 4 asks for asynchronous messaging by name, and this picture is what capstones in this course tend to grow into, one plate at a time. When yours does, you’ll already know what every arrow costs.

08Self-Check

Six quick questions. Pick an answer to see whether you’re right and why. Nothing is recorded; this is for you.

09Before Session 21 — November 16

Three things, none longer than a coffee. The list remembers your progress on this device.

• Drive Plate 2.1 until you can narrate it: send a rush, kill the worker mid-drain, revive it — and say in one sentence where the notes were while the worker was dead, and why “Lost” never moved. Expect to drive it in Session 21.
• Drive Plate 5.1 to both endings: a duplicate charge with keys off, and exactly one charge at 100% crash chance with keys on. Then say which sentence of §05 the second ending demonstrates.
• Keep Project 3 moving. You are closer than it feels — the first time the teardown-and-rebuild runs clean, you’ll want to show somebody, and Session 21 is full of somebodies.

10Key Terms

Synchronous call

The phone call: A asks B and waits on the line. A’s fate is chained to B’s for the length of the call.

Message

The note: a small written ask (“send a receipt for order 741”) that can wait, travel, and be re-read.

Queue

The waiting line of notes, oldest first. A shock absorber: spiky arrivals in, steady work out, backlog in between.

Producer / consumer

The service that leaves the note, and the worker that pulls notes and does them.

Work queue

The diner’s ticket rail: each note done by exactly ONE worker. Line too long? Hire another cook.

Broadcast (topic, pub/sub)

The PA system: one announcement, every subscriber gets their own copy to act on.

Event

A fact, announced: “order 741 happened.” Not an instruction to anyone in particular.

Event-driven architecture

Services announce facts and subscribe to facts. New departments join without the announcer changing.

Acknowledgment (ack)

The worker’s “done” back to the queue — the only reason the queue ever lets go of a note.

At-least-once delivery

The only promise physics allows: every note arrives, and occasionally arrives twice. Plan for the twice.

Redelivery

The queue, hearing silence instead of an ack, hands the same note out again — correctly assuming the worst.

Idempotency

Doing it twice leaves the world exactly as doing it once. The thermostat SET to 68, not raised by 3.

Idempotency key

The ID on the note — “the charge for order 741” — that lets a worker recognize work it has already done.

Done-list

The record of finished IDs, checked before working. Durable, shared state — the landlord’s database, not a worker’s memory.

Retry

Trying again after a failure — the polite way to insist. Requires idempotency, always.

Backoff

Waiting 1s, 2s, 4s… between retries — air for a struggling service. Notes 08’s “never restart harder,” re-worn.

Jitter

A pinch of randomness in retry waits, so a thousand retriers arrive as a drizzle instead of a stampede.

Dead-letter queue

The needs-a-human tray: where a note goes after its last failed retry. Retries repair accidents, not mistakes.

SQS / Pub/Sub / Service Bus

The managed queue-and-topic rentals on AWS, Google, and Azure. Same note, same door.

Kafka

The heavy-duty flavor: a log everyone reads at their own bookmark. For now, that sentence is enough.

No term matches that filter.