Is the Park Healthy?

01You Can’t Walk the Park at Night

Last week ended on a question, and it has been hanging in the air for seven days: the park is written down, the robot rebuilds it on demand, nothing reaches production except through a file that was reviewed… so the park exists. But is it healthy — right now, tonight, while you’re asleep?

If you owned a real park, you’d walk it. Jiggle the doors, listen for a rattle in the machinery, smell for smoke. You cannot do that here. Your park is rented rooms in a building you will never enter, in a city you have probably never visited. There is no flashlight to carry down the rows at midnight. Whatever you know about the park’s health, you know through instruments — or you don’t know it at all.

Doctors solved this problem centuries ago, because they can’t open a patient up just to check on them. They read a pulse — a few numbers, taken over and over. They ask for the story — “tell me exactly what happened, and when.” And for the hardest questions they inject a dye and watch where it goes. Three instruments, three different kinds of answer, one patient.

Observability is exactly that, for systems: the park wired up so you can tell what is happening inside without opening anything up. The three instruments even keep their jobs — they’re called metrics (the pulse), logs (the story), and traces (the dye), and this week is one section on each. Then the fourth wire, the one that makes the other three matter at 2 a.m.: the alarm, connected to a sleeping human. And finally the humans themselves — the people who carry the pager, and the surprisingly humane arithmetic they run on.

02Logs — The Diary

Start with the oldest instrument. A log is a diary: every service in the park writes down what happened to it, one line at a time, the moment it happens. “Receipt 1042 charged $20.” “Card declined.” “Note left on the door.” Nobody reads the diary on a good day. On a bad day it is the only eyewitness you have.

One habit separates a useful diary from a useless one, and it costs you nothing to adopt today: write lines a machine can also read. “payment: card declined for receipt 1042 (212ms)” is fine for a human — but a machine can only squint at it. Give the same facts names, and suddenly they can be counted, filtered, and graphed. That’s a structured log, and it usually just means JSON:

{
  "time":    "2026-11-23T21:04:18Z",  ← when, exactly
  "service": "payment",              ← which room wrote it
  "level":   "error",                ← how bad (info / warning / error)
  "msg":     "card declined",        ← what happened, for humans
  "receipt": 1042,                   ← which exact order, for machines
  "took_ms": 212                     ← how long it took
}

Now the problem that arrives with success. One shop, one diary — easy. But your park has many rooms now: the shop front, the payment service, the database, the workers behind Notes 11’s door, and the manager spinning copies up and down. A real park produces a million diary lines an hour, scattered across machines that the manager creates and destroys without asking you. Logging into one machine and running a search there is a flashlight in a landfill — and the machine holding the line you need may not even exist anymore.

So every real park does one thing: every diary streams, as it is written, into one shared diary for the whole park — a central, searchable place. This is called central log collection, and it turns “which of forty machines saw receipt 1042?” into one search box: receipt:1042, every room’s entries, in time order. That’s also why the structure matters: you can only type receipt:1042 because the line has a field called receipt.

03Metrics — The Gauges

The diary answers “what happened to receipt 1042?” It is terrible at answering “how are we doing, generally?” — you don’t take a patient’s pulse by rereading their biography. For that you want the second instrument: metrics, which are just numbers, measured over and over, kept as a line through time. Requests per second. Errors per hundred requests. How long the typical request takes. How many notes are waiting on the door (Notes 11’s queue depth). How hard the machines are working.

Metrics are the cheap instrument, and that’s their superpower. A log line is a sentence per event; a metric is one number per second no matter how busy the park is. Cheap to keep means you keep months of them, and months of numbers make charts — and a wall of those charts is a dashboard: the park’s instrument panel, the thing on the big screen in every operations room you will ever visit.

The average is a liar — meet p99

Here is the one piece of arithmetic this section needs, and it matters because the obvious summary — the average — hides pain. Suppose 100 customers check out. Ninety-nine wait 1 second. One waits 60 seconds. The average wait is about 1.6 seconds — a number that says everything is fine. Meanwhile one customer just stared at a spinner for a full minute.

So instead of averaging, line everyone up from fastest to slowest and read positions. The wait at the halfway point is the p50 — the typical customer. The wait at position 99 of 100 is the p99 — the slowest 1%. In our shop: p50 says 1 second, p99 says 60. Same hundred customers, and now the suffering shows. That slow 1% is called the tail, and real teams obsess over it for an unsentimental reason: the customers with the biggest carts, the most photos, the heaviest use — the ones you least want to lose — are exactly the ones who hit the slow paths most often. The tail is where your best customers live.

04Traces — The Dye

Think about what happens when one customer buys one photo print in the park you’ve built this term: the shop front takes the order, calls payment, writes to the database, leaves a note on the door of the receipt queue, and a worker picks the note up and does the slow part. Five rooms, one purchase. Now the request is slow. Each room’s diary says “I was fine.” Each room’s gauges look normal. The slowness lives in the journey — and nobody owns the journey.

So inject dye. At the front door, every request gets stamped with a random ID — the trace ID — and every room passes the stamp along like a relay baton, writing down “I held this request from here to here.” Each room’s slice is called a span. Collect the spans that share a stamp, sort them on a timeline, and you get the classic waterfall: the whole journey of receipt 1042, drawn as bars — including the part that happened after the customer stopped watching, because the dye follows the receipt, not the customer. One glance at the waterfall answers the question nothing else could: the longest bar is where the time went.

Try it. Press new request, read the waterfall, and click the span you think is the bottleneck — the bar, or its button below. Then press send 100 requests a few times and watch the two numbers on the right: p50 barely moves, while p99 climbs as the occasional disaster lands in the history. You are watching the tail form, live.

Notice what your eye learned after three or four requests: you stopped reading numbers and started looking for the long bar. That’s the entire skill. Real tracing tools (Jaeger, Zipkin, X-Ray, Cloud Trace — any of them) draw exactly this picture for real requests, and engineers debug multi-service slowness by doing what you just did: follow the dye, find the long bar, go knock on that room’s door.

05Alarms That Wake Humans

Gauges, diaries, and dye share one weakness: someone has to be looking. At 2 a.m. nobody is. So the last wire runs from the instruments to a sleeping human: an alert is a rule — “if this gauge crosses this line for this long, wake someone” — and the pager it rings is the park’s smoke alarm.

Which gauge do you wire it to? This is the one place this week where beginners reliably get it backwards, so it gets the box:

And one honest sentence about the failure mode, because every team rediscovers it the hard way: a pager that cries wolf gets ignored. Wire alarms to every twitchy machine-gauge and the on-call human learns, within weeks, to silence the phone and roll over — and then the real fire burns while a hundred false alarms scroll by. This is called alert fatigue, and the cure is discipline, not volume: every alarm must be urgent, customer-shaped, and actionable — or it shouldn’t page anyone.

Which is why mature teams sort every alarm into exactly two buckets:

06The People With the Pager — SRE

Someone’s phone is on the other end of those wires. At most companies that someone has a job title: site reliability engineer — SRE. The idea, born at Google and copied everywhere, is that keeping the park healthy is an engineering problem, worked by engineers, with budgets and arithmetic — not a heroic all-nighter culture. Three SRE ideas are worth carrying out of this course, and each fits in a breath.

Idea 1 — The SLO: a promise with a number

“The park should be reliable” is a wish. An SLO — service level objective — is a promise you can measure: “99.9% of requests succeed.” Notice what the promise deliberately is not: 100%. Notes 10 taught you why — machines die, regions have bad days, perfection is a lie that only costs more the harder you chase it. So you promise almost perfect, and the “almost” has a size. Using a 30-day month — 43,200 minutes, a convention we’ll keep because the arithmetic comes out clean — 99.9% leaves you 43.2 minutes of broken per month. Promise 99% and you get 432 minutes (7.2 hours). Promise 99.99% and you get 4.32 minutes — barely time to read the page before you’ve blown it.

Idea 2 — The error budget: permission to break things, in writing

Here is the genuinely clever part. That gap between your promise and perfection — the 43.2 minutes — is not shame. It is an allowance, and you are supposed to spend it: on bold deploys, on experiments, on moving fast. Budget left? Ship. Budget gone? Stop shipping features and work on reliability until the month resets. And notice what that rule just did: it ended the oldest war in software — the product team shouting “ship it!” at the operations team shouting “don’t break it!” — and replaced the shouting with arithmetic both sides agreed to in advance. Nobody argues about feelings; everybody reads the meter.

Drive it to the edge once, on purpose: at 99.9%, five 10-minute incidents fit with change to spare — a sixth flips the sign. Then switch the promise to 99.99% and watch almost any slider position flip it: tighter promises buy trust and cost freedom, and that trade — not a technology — is what your capstone’s reliability story should be about.

Idea 3 — The blameless postmortem

After every real incident, the team writes a short fire report: what happened, minute by minute, and what will change so it can’t happen the same way twice. The non-negotiable rule is in the name — blameless. The report asks “what let this happen?”, never “who did this?” — and not because anyone is being gentle. It’s because the alternative doesn’t work: punish the engineer who broke production and you teach every engineer to hide the next near-miss — and a scared engineer hides the next smoke. The person who typed the fatal command is almost never the problem; the system that let one tired human sink the park with one command is. Fix the system. Thank the human for the details.

07The Whole Park, Wired

Step back and look at what fifteen weeks built. The shop front from your first project. The queue with notes on its door, and the workers behind it. The database, the bucket of photos. The manager, keeping the right number of copies alive. The robot, holding the whole park in a text file it can rebuild from nothing. And now, laid over all of it, this week’s layer: a gauge on everything, every diary flowing to one place, dye through every request, and one wire running to a human who trusts the alarm enough to get out of bed.

That picture is also the quiet definition of this course’s last word, observable, from the capstone spec: not “it has a dashboard somewhere,” but when it breaks at 2 a.m., the instruments — not luck — tell you what broke, where, and for whom.

08Self-Check

Six quick questions. Pick an answer to see whether you’re right and why. Nothing is recorded; this is for you.

09Before Session 25 — November 30

One real deadline this week, and it’s a small one on purpose. The list remembers your progress on this device.

• Capstone proposal — due this week. One paragraph, submitted per the course page. It needs exactly two things: what you’ll build, and which pieces of the semester it uses — a queue? the manager? the robot? which gauges will it wear? No template, no diagram, no cost table. A paragraph that lets me say “yes — go.”
• Drive Plate 4.1 until you can do two things without thinking: spot the bottleneck in one glance, and explain in one sentence why p99 kept climbing while p50 barely moved. Expect to narrate a waterfall in Session 25.
• Spend an error budget on Plate 6.1: find the exact number of 10-minute incidents that flips a 99.9% promise from SHIP to STOP SHIPPING, then say out loud what the team has to do next. (Hint: it isn’t “argue.”)
• Pick your capstone’s one page-worthy alarm. Write the rule in the Plate 7.1 style — “IF symptom FOR minutes → PAGE” — and check it against the golden rule: would a customer notice? Bring it to Session 25.

10Key Terms

Observability

The park wired so you can tell what is happening inside without opening anything up — including answering questions you didn’t plan for.

Logs

The diary: every service writes what happened, one line at a time. The only eyewitness on a bad day.

Structured log

A diary line a machine can also read — named fields (usually JSON), so entries can be counted, filtered, and searched.

Central log collection

Every diary streaming into one shared, searchable place. The cure for “which of forty machines saw receipt 1042?”

Metrics

The gauges: numbers measured over and over, kept as lines through time. Cheap enough to keep months of.

Dashboard

A wall of gauge charts — the park’s instrument panel.

Percentile (p50 / p99)

Line all waits up fastest to slowest and read positions: p50 is the typical customer, p99 the slowest 1%. The honest latency numbers.

Tail latency

What the slowest few percent experience. Averages hide it; your heaviest (best) users live in it.

Trace

The dye: one request’s whole journey, stitched across services by a shared trace ID and drawn as a waterfall.

Span

One room’s slice of a trace — “I held this request from here to here.” The longest bar is where the time went.

Alert

A rule wiring a gauge to a human: “if this crosses this line for this long, tell someone.”

Symptom vs. cause

What the customer feels (errors, latency) vs. what the machine feels (CPU, disk). Page on symptoms; causes are clues.

Page vs. ticket

Wake someone at 3 a.m. vs. fix it Monday. Every alarm belongs to exactly one bucket.

Alert fatigue

A pager that cries wolf gets ignored — and then the real fire burns. Cured by fewer, customer-shaped, actionable alarms.

SRE

Site reliability engineering: running the park treated as an engineering discipline, with budgets and arithmetic instead of heroics.

SLO

A promise with a number — “99.9% of requests succeed.” Deliberately not 100%, because perfection is a lie with a price tag.

Error budget

The gap between the promise and perfect (43.2 min/month at 99.9%), spent on moving fast. Budget empty → stop shipping, fix reliability.

Blameless postmortem

The fire report that asks “what let this happen?”, never “who?” — because a scared engineer hides the next smoke.

No term matches that filter.