The Manager You Hire

01The 2 a.m. Problem

Be honest about what Project 2 has made you: a babysitter. You start the database first, because the API falls over without it. Then the API, the worker, the web front end. When the worker crashed on Tuesday, you noticed on Wednesday and typed a restart. And when we asked for horizontal scaling, you started extra copies by hand and told the front desk about each one.

Every one of those orders is easy. The job of giving them is not. Four containers on one machine is an evening. Thirty on ten machines is a career. Three hundred is impossible, because containers crash at 2 a.m., machines die mid-afternoon, and traffic spikes while you’re in class. Notes 07 ended on exactly this cliff: by hand is fine for a laptop and hopeless for a fleet.

Notice the shape of the problem. No single order is hard. It’s that somebody must keep giving them, forever, at all hours, without mistakes — and that somebody should not be you. So this week you hire a manager — an orchestrator — and the most famous one is Kubernetes. (People write it K8s: a K, eight letters, an s.)

02Declare, Don’t Command

Here is the one idea the whole week stands on. Until now you have been giving orders: start this, stop that, restart the sick one. An order runs once and is forgotten. If the world changes afterward — a crash, a dead machine — the order doesn’t care.

Kubernetes works the other way. You hand it a wish list: a written description of the world you want. “Three copies of the web container, always. Two of the API. One worker.” Then the manager does something beautifully simple, every few seconds, forever:

1. Look at the live world. What is actually running right now?
2. Compare it to the wish list. Any difference?
3. Fix the difference. Too few copies? Start one. Too many? Stop one. A machine vanished with two copies on it? Start two elsewhere.

You have met this reflex: target tracking in Notes 03 was a thermostat for instance counts. Kubernetes is the thermostat applied to everything — copies, versions, traffic — and nearly every feature this week is this one loop wearing a different costume.

The consequence that matters: the fixing never stops. An order runs once; a wish is enforced continuously. When your container crashes at 2 a.m., nobody pages you — within seconds the loop sees wish ≠ reality and starts a replacement. You didn’t handle the failure. You made failure boring. That is the entire pitch; the rest is detail.

03The Cast, in Plain Words

Kubernetes has a famously large vocabulary. You need four words of it; everything else this week is built from these.

Why pods, not bare containers? Occasionally two containers only make sense as a unit — your app plus a tiny helper shipping its logs — and the promise they need is “one address, scheduled together, dead together.” A pod is that promise, boxed. Nearly every pod you make holds one container, so read “pod” as “a running container, plus paperwork.” One naming habit: the wish says replicas: 3, and each copy is a replica — the same word as storage in Week 4, for the same reason: identical copies make losing one a shrug.

04The Loop, Live

Enough description. Below is a toy cluster: three machines, one wish, the loop running once a second and narrating what it sees. Your job is to make the world stay wrong. You will fail, and the ways you fail are the lesson.

Four experiments, in this order:

1. Kill a pod. Reality drops to 2; within a second the manager says so and starts a replacement. You attacked reality, and the wish never changed — so reality got repaired. The only way to really remove a pod is to change the wish.
2. Slide the wish to 0, then 8. Both are legal wishes, calmly enforced. Zero is how you turn a service off declaratively.
3. Make one crash repeatedly. Watch the restart waits: 1s, 2s, 4s, 8s… The manager never gives up, but it backs off. Section 07 explains why this crash loop is a message to you, not a job for the manager.
4. Drain machines. One drain: the pods move house. Drain all three: the pods go pending — the manager cannot conjure hardware. (Section 08 hires the robot that expands the building.)

05A Permanent Name for a Changing Crowd

The loop creates a new problem; feel it before fixing it. Your web pods call your API pods — but API pods are cattle now: they die, respawn, move between machines, and every new pod gets a new address. Any address you write down is a note about the past. Who do you even call?

Notes 05 solved this at city scale: the front desk is the one thing with a permanent name, while the anonymous servers behind it come and go. Kubernetes repeats the trick inside the cluster and calls it — confusingly but memorably — a Service: a permanent in-cluster name (api) with a tiny front desk behind it, forwarding each call to whichever pods currently qualify.

Qualify how? Labels. A label is a name tag stuck on a pod — app=api — and the Service’s selector says “route to everyone wearing this badge, whoever that is right now.” No address list anywhere; the deployment stamps the badge on every copy it makes, so replacements join the moment they’re ready. In Compose last week, your API reached the database at the name db and it just worked; a Service is that comfort, grown up and crash-proof.

Traffic from outside — your actual users? Mark a Service as type LoadBalancer and Kubernetes asks the provider for a real front desk — the very load balancer of Notes 05, public name and all — wired to the matching pods. The full journey: provider’s front desk → Service → whoever wears the badge this minute. (When one cluster hosts many sites behind one desk, a routing layer called an Ingress picks the Service by address and path — know the name, skip the details this semester.) And notice why a sticker beats an address list: the selector is evaluated continuously, against the present. A list describes who existed when you wrote it; “everyone wearing the badge” describes whoever exists right now.

06Badge Check, Live

Here is that Service as a toy. Five pods: three born wearing the app=web badge, two wearing app=worker. The Service routes only to app=web. Click any badge to re-label a pod and watch the routing follow the stickers — then kill a pod mid-stream and see why nobody calling web ever notices.

Three things to catch. Re-label a worker as web and it receives web traffic immediately — routing follows stickers, not history. Strip every web badge and the name still exists but calls fail — a name is not a guarantee that anyone qualifies. And after a kill, the replacement arrives already wearing the badge but gets no traffic until it’s ready — the next section’s subject.

07Alive vs. Ready

The manager restarts sick containers and routes around busy ones. How does it know? It asks. You built the answer last week: every Project 2 service has a /healthz endpoint. Kubernetes calls such a question a probe, and it asks two different ones, because “sick” comes in two kinds:

• Liveness — “are you alive?” Asked every few seconds, forever. Fail a few in a row and the manager concludes the container is wedged — running but beyond saving — and restarts it. The 2 a.m. restart you used to do by hand, now a reflex.
• Readiness — “ready for customers?” Fail this and nothing restarts — you’re just taken off the Service’s list until you pass again. A pod warming up, or waiting on its database, is alive but not ready. Exactly the front desk’s health check from Notes 05, one layer down.

Keep them straight by their cures: liveness failure → replace; readiness failure → don’t seat anyone there. Notes 03 called these the two reflexes — replacing failure and routing around it — and Kubernetes runs both for every pod you own. So your probes must be honest: a /healthz that always says “ok” turns both reflexes off at the worst moment — and waves broken pods into a rolling update (§08).

08Two Ways to Grow (and One Way to Change)

Scaling in Kubernetes is two knobs, and you already know both robots that turn them.

• More pods — the horizontal pod autoscaler. The hiring manager from Notes 03, re-hired: watch a measure of busy-ness (CPU is the lazy default) and, when the waiters are run off their feet, raise the deployment’s replicas wish. Note the elegance: the autoscaler doesn’t start pods. It edits the wish, and the ordinary loop does the rest. A loop adjusting the number another loop enforces — costumes all the way down.
• More machines — the cluster autoscaler. Pods need somewhere to stand. When new pods go pending because no node has room (you caused this in Plate 4.1), the cluster autoscaler rents another machine and adds it to the pool; when nodes sit near-empty, it drains and returns them. Hiring waiters vs. expanding the building — both automatic, and both metered: more nodes, more bill.

And version upgrades — the scary Friday deploy — become choreography. Change the image in the wish from :1.4.0 to :1.5.0 and the deployment performs a rolling update:

The choreography in words: a new-version pod starts and must pass readiness before it gets a single request; only then is one old pod drained — seated last, allowed to finish its diners, then removed (Notes 03’s connection draining); repeat until the crowd has turned over. A broken new version never passes readiness, so the rollout simply stalls with the old version still serving — and one command rolls the wish back. No gap, no magic: readiness gates plus one-in-one-out.

09Do You Actually Need Kubernetes?

Honest altitude check, because this is the most over-prescribed tool in the industry. Kubernetes was built by Google-scale people for Google-scale problems: thousands of containers, dozens of teams, one shared pool of machines. Its price is real — a new vocabulary, a stack of YAML, a cluster that itself needs upgrading and securing. You are not Google. Neither is almost anyone who runs Kubernetes anyway.

And Project 2? Read the spec’s words: an orchestration platform, with health checks and horizontal scaling. Compose on a cloud VM counts. ECS counts. Cloud Run counts. Kubernetes counts. What’s graded is not the logo; it’s whether your system states wishes something else enforces — and whether you can explain what happens when each piece dies. We teach the Kubernetes concepts because they are the shared language; everything above is a dialect of them.

10The Wish List, Written Down

One thing remains: what a wish physically is. A small text file, in YAML — the same indentation-is-structure format as last week’s compose.yaml. Kubernetes calls it a manifest, handed over with one command: kubectl apply -f wish.yaml (“make the world match this file”). A complete, real pair, every line translated:

apiVersion: apps/v1 # which page of the form catalog
kind: Deployment # the kind of wish: keep copies alive
metadata:
  name: web # the wish’s own name
spec:
  replicas: 3 # the wish itself — three, always
  selector:
    matchLabels:
      app: web # “my pods wear this badge”
  template: # the pod to make copies of
    metadata:
      labels:
        app: web # every copy born wearing it
    spec:
      containers:
      - name: web
        image: registry.example.com/photos-web:1.4.0 # the frozen meal, by reference
        ports:
        - containerPort: 8000 # the app listens here
        readinessProbe: # “ready for customers?”
          httpGet: { path: /healthz, port: 8000 }

apiVersion: v1
kind: Service # permanent name + tiny front desk
metadata:
  name: web # in-cluster callers use http://web
spec:
  selector:
    app: web # whoever wears the badge, right now
  ports:
  - port: 80 # the name’s door…
    targetPort: 8000 # …forwarded to each pod’s door

Read line 17 with last week’s eyes: the wish names an image by reference, and the cluster pulls it from your registry on every pod start. This is where Plate 7.1’s lesson lands — point a wish at a movable nickname like :latest and your “identical replicas” can quietly run different bits. Unique tag or digest, always. One sentence to carry out: the file is not a script. Nothing in it is a command; it describes an end state, and the loop earns its keep making it true — today, and at 2 a.m. in November. (Week 12 pushes this to everything: whole clouds described in files. Same religion, bigger church.)

11Self-Check

Six quick questions. Pick an answer to see whether you’re right and why. Nothing is recorded; this is for you.

12Before Session 15 — October 26

Project 2 is due at the end of this week. You are closer than it feels: the images exist, the system runs on your laptop, and these notes named everything that’s left. Work the list top to bottom — it is drawn from the project description and from how I grade. It remembers your progress on this device.

• All three services plus the managed datastore run under your orchestration platform — Kubernetes, ECS, Cloud Run, or Compose on a cloud VM (§09: the logo is not graded). A fresh deploy from scratch works.
• Every image lives in your cloud registry with a unique tag — not latest — and the deployment references those. Digests recorded in the report.
• The datastore password is in no image and no repo. It arrives at run time, from the secrets manager, through a role. Run the Notes 07 image checklist per service.
• Health checks demonstrably work. Kill one container on purpose and capture the restart — a log line, a screenshot, a timestamp pair. “I configured it” is not evidence; the restart is.
• Horizontal scaling demonstrated. Scale one service from one copy to three (by rule or by hand — both count) and show the copies sharing traffic. State the wish before and after.
• The failure-mode write-up covers each service, datastore included: what happens when it dies, what the platform does, what users see — and whether it’s an accident (self-healing fixes it) or a mistake (a crash loop; §07).
• The report defends the architecture — the cost, security, and reliability consequences of your choices. “It works” is the floor.
• Submit the repository link and the report via Brightspace by the end of the week. The commit history should show the work happening, not one heroic Friday push.

13Key Terms

Orchestrator

The manager you hire: software that keeps containers matching a stated wish.

Kubernetes (K8s)

The industry-standard orchestrator. “K8s” = K, eight letters, s.

Declarative

Stating the world you want instead of the steps to get there. The opposite of giving orders.

Reconcile loop

Look, compare to the wish, fix the difference — every few seconds, forever.

Cluster

A pool of machines handed to the orchestrator to fill.

Node

One machine in the cluster — an instance from Notes 02–03, now anonymous staff.

Control plane

The manager’s office: holds the wish list, runs the loop, runs none of your containers.

Pod

The smallest unit: one container (rarely a few sharing an address, dead together), plus paperwork. Disposable by design.

Deployment

The wish “keep N copies of this pod alive, and roll new versions out gently.” Each copy is a replica.

Manifest

A wish written down: the YAML file you hand over with kubectl apply.

Label

A name tag stuck on a pod, like app=web.

Selector

“Everyone wearing this badge, whoever that is right now.” Never a list of addresses.

Service

A permanent in-cluster name with a tiny front desk, routing to whichever pods currently match its selector.

LoadBalancer service

A Service that also asks the cloud for a real public front desk (Notes 05).

Ingress

Shares one front desk among many Services, by address and path. Name only, this semester.

Liveness probe

“Are you alive?” Fail repeatedly and the container is restarted.

Readiness probe

“Ready for customers?” Fail and you’re off the Service’s list — no restart, no traffic.

CrashLoopBackOff

Restarts of a container that keeps dying, with doubling waits. Read the logs; don’t restart harder.

Horizontal pod autoscaler

A loop that edits the replicas wish from load — the hiring manager, re-hired for pods.

Cluster autoscaler

Rents and returns machines as pods need room (no room = Pending). Expands the building itself.

Rolling update

New pods pass readiness in; old pods drain out, one-in-one-out. Zero downtime as choreography.

EKS / GKE / AKS

Managed Kubernetes on AWS, Google, Azure: the control plane run for you, the wishes still yours.

No term matches that filter.