Compute That Follows the Load

01Reading an Instance Type Name

Last week ended with each of you holding the name of the smallest instance your provider rents. This week starts with the catalog those names come from, and the first encounter is intimidating on purpose: AWS alone lists several hundred instance types. The saving fact is that the catalog is not a list, it is a grammar. Every name encodes the same three decisions — a family, a generation, and a size — and once you can parse them, eight hundred products collapse into a dozen ideas.

The family is the interesting decision, and it is really a statement about ratios. Physical hosts are built with CPU, memory, disk, and network in fixed proportion; families are the menu of proportions. General purpose is the balanced default. Compute-optimized skews toward CPU for the same money; memory-optimized skews toward RAM; storage-optimized ships with fast local disk; accelerated families carry GPUs. Choosing a family is declaring which resource your workload will run out of first — and if you do not know yet, that is what the general-purpose family is for.

Now the grammar itself. Pick a provider, a family, and a size below, and read the name token by token. The three providers encode the same decisions with different syntax — AWS makes you learn size words, Google and Azure put the vCPU count right in the name.

Two rules fall out of the decoder, and both matter for design. First, within a family, everything scales linearly: each size step doubles vCPU, memory, and — almost exactly — price. Two xlarge cost what one 2xlarge costs. So there is no bulk discount for buying big, which means the interesting choice is not “how big a machine” but “how many small ones” — more small instances can be added and removed in finer steps and spread across availability zones. Everything in Section 04 rests on that premise. Second, prefer the current generation: newer generations are almost always faster and cheaper per unit of work, so the generation digit is the one place where the safe choice and the cheap choice agree.

One family deserves a caution flag: the burstable types (AWS t, GCP e2 shared-core, Azure B) are the cheapest line in every catalog because they assume you will mostly be idle — they accumulate CPU credits while quiet and spend them in bursts. Perfect for a course project or a low-traffic site; quietly disastrous under sustained load, when the credits run out and the instance throttles to a crawl. Know which one you are renting. And remember from Notes 02 what a vCPU is — a scheduled hardware thread, not a core — so benchmark before you trust a size, and resize afterward: changing an instance type is a stop-change-start operation measured in minutes, a luxury the 2004 procurement cycle could not imagine.

02Machine Images: Golden or On-Boot

Notes 02 defined the machine image in passing: the template a boot disk is created from. Now it becomes a design decision, because the image determines what an instance is the moment it boots — and, more importantly for this week, how long it takes to become useful. The vocabulary: an AMI on AWS, a custom image on Google Cloud, a managed image (published through a Compute Gallery) on Azure. Providers ship stock images for every mainstream OS; marketplaces sell pre-configured ones; and you can capture your own from a configured instance or build them in a pipeline.

When an instance launches, the image becomes the boot volume, the OS starts, and then first-boot configuration runs — the user data script on AWS and Azure, the startup script on GCP, most of it standardized by a tool called cloud-init. That gives you a spectrum with two ends, and where you sit on it is a real architectural choice:

The golden-image discipline has a name worth knowing: immutable infrastructure. You never patch a running instance; you build a new image, roll new instances, and destroy the old ones. If that sounds familiar, it should — it is exactly the philosophy of container images from Notes 02, applied at VM granularity, and it is the operational meaning of “cattle, not pets.” A server you would hesitate to delete is a server you cannot trust yourself to rebuild.

In practice mature teams take the middle path: bake the heavy, slow-changing things into the image — runtime, agents, dependencies — and inject only the last mile at boot: configuration values, secrets, the pointer to the current application version. Why this section sits here, between type names and autoscaling, is the punchline to carry forward: boot time is scaling latency. An autoscaler can react to a traffic spike no faster than a new instance can become useful, and a four-minute configure-on-boot script means four minutes of overload every time demand jumps. Keep that number in your head; you will watch it matter in Plate 4.1.

03Just Enough Load Balancing

Everything in the next section changes the number of instances behind your service, sometimes several times an hour. Clients cannot be expected to chase a changing set of addresses, so fleets get a front door: a load balancer — one stable name and address that accepts every request and spreads them across whatever instances are currently registered behind it. With a load balancer in front, instances become anonymous. They can be added, removed, and replaced without anyone outside the fleet noticing, which is precisely the property autoscaling needs.

The load balancer's second job matters as much as its first: it decides who deserves traffic. A health check is a probe — typically an HTTP request to something like /healthz every ten seconds or so — and an instance that fails a few in a row is quietly taken out of rotation until it recovers. The same signal feeds the autoscaling group, which goes one step further: an instance that stays unhealthy gets terminated and replaced from the launch template. Notice these are two different reflexes — routing around failure and replacing failure — and together they are why a fleet of disposable cattle can offer better uptime than one lovingly maintained pet. Self-healing is not magic; it is a probe, a timeout, and a template.

That is deliberately all we take this week. Load balancers have a rich life of their own — the layer-4 versus layer-7 distinction, TLS termination, how traffic reaches them across zones, DNS, CDNs in front of everything — and all of it is Week 5's subject, where the names get specific (AWS ALB/NLB, Google Cloud Load Balancing, Azure Load Balancer and Application Gateway). Until then, treat the load balancer as a sealed box with one contract: it makes N interchangeable instances look like one reliable one.

04Autoscaling: The Feedback Loop, Realized

Notes 01 promised that when capacity arrives in seconds and bills by the second, forecasting stops being a bet and becomes a feedback loop. This section is that loop with part numbers. An autoscaling group (AWS), managed instance group (GCP), or VM scale set (Azure) is three things bundled: a launch template — the instance type, image, and settings from Sections 01–02, i.e., the recipe for one more server; a size envelope — minimum, desired, and maximum instance counts; and policies that move “desired” automatically. Once the group exists, instance count stops being a decision anyone makes and becomes an output of policy. That sentence is the week.

Policies come in three useful flavors. Target tracking is a thermostat: name a metric and a setpoint — “keep average CPU at 60%” — and the group adds and removes instances to hold it. It is the right default. Step scaling fires fixed adjustments at thresholds, for when you want explicit control of the response curve. And scheduled scaling handles the load you can put on a calendar: scale to eight at 8:45 every weekday morning, back to two at seven. Recall the Black Friday retailer from Notes 01 — the fleet that sat idle 360 days a year. Under scheduled scaling, Black Friday is a calendar entry. (The providers also offer predictive modes that learn the calendar for you; treat them as scheduled scaling with a forecasting model attached.)

Walk the scale-out chain once, slowly, because every link is something from earlier in these notes: the metric crosses the line → the policy raises desired → the group launches instances from the template → each one boots and runs its first-boot configuration (Section 02 — this is where your image strategy is either seconds or minutes) → it passes health checks → the load balancer registers it (Section 03) and traffic flows. Scale-in runs the film backward with two courtesies: the group picks victims to keep zones balanced, and connection draining lets in-flight requests finish before termination. And note the asymmetry every experienced operator builds in: scale out aggressively, scale in conservatively. Running a few extra instances for ten minutes costs cents; shedding capacity into a rising load drops requests on the floor.

Three experiments to run before Session 4, in this order:

1. Manufacture thrashing. Set cooldown to 0 and target to 90%. The wobble in the demand curve now drives launch–terminate–launch cycles — watch the action count climb while overload minutes barely improve. That oscillation is called thrashing (or flapping), and the cure you just removed — a cooldown, or a wider band between scale-out and scale-in thresholds — is hysteresis, the same idea that keeps your thermostat from cycling the furnace every ninety seconds.
2. Price a slow image. Put the target back to 65, cooldown to 10, and drag boot + warm-up to 16 minutes — a stock image with a heavyweight configure-on-boot script. Watch the morning ramp: the group decides correctly and still serves overload for a quarter hour, while the dashed line reminds you the meter runs from launch, not from readiness. Now drag warm-up to 2 — a golden image — and watch Section 02 pay for itself.
3. Cap the fleet. Set max to 4 and watch the evening spike flat-top against it. The max is your budget guardrail, and this is what it looks like when it binds: autoscaling cannot buy what you told it never to buy. Whether that plateau is a prudent cost ceiling or an outage depends entirely on what those requests were worth — which is a Notes 01 question, not a technical one.

Two limits to keep in view, so autoscaling stays a tool and not a superstition. It scales the stateless tier: if every instance leans on one database, ten more instances just crowd the same bottleneck (Week 9 takes that on). And it picks the metric you tell it to watch — CPU is a fine proxy for CPU-bound work and a lie for everything else; request count per target or queue depth is often closer to the truth. Scale on the resource you actually run out of.

05Serverless Functions: Scaling by Request

Take the autoscaling group and push every parameter to its limit: minimum zero, scale-out in milliseconds, and the unit of capacity shrunk from “an instance” to “one request.” Then hand the whole apparatus to the provider to operate. That is functions as a service — AWS Lambda, Google Cloud Run functions, Azure Functions — and it is why FaaS sat at the far end of the responsibility stack in Notes 01. You supply two things: a function, and the events that should invoke it — an HTTP request, an object landing in storage, a message arriving on a queue, a timer. Everything between the event and your first line of code is the provider's problem.

What actually happens between them is machinery you already know. When an event arrives, the provider routes it to a warm execution environment — in the major implementations, a microVM of exactly the kind Notes 02 ended on; Lambda runs on Firecracker. If a warm one exists, your code runs in single-digit milliseconds. If not, the provider must create one — boot the microVM, load the runtime and your code, run your initialization — and that pause, somewhere between a hundred milliseconds and several seconds, is the famous cold start. Load rises, more environments get created; load falls, they are reclaimed; load hits zero and you are running — and paying for — nothing at all. Scale to zero is the one thing no instance fleet can do, because an autoscaling group's minimum useful size is one booted OS, while a function's is nothing.

Cold starts deserve engineering judgment rather than fear. For queue processing and event glue, nobody notices a 400 ms cold start. For an interactive API where tail latency is the contract, they can be disqualifying — and the mitigations are instructive: trim dependencies and pick lighter runtimes (smaller things load faster), or pay for provisioned concurrency — environments kept warm on retainer. Read that last one through Notes 01 eyes: you are paying for idle capacity to buy latency, which is exactly the trade the elastic model was supposed to dissolve. There is no free lunch here, only a well-priced menu.

The constraints are the flip side of the execution model, not arbitrary rules. Environments are created and destroyed at the provider's whim, so functions must be stateless — anything worth keeping goes to external storage (Week 4) or a database (Week 9). Executions have hard time caps (about 15 minutes on the major platforms), so long-running work belongs elsewhere. And because each environment handles one request at a time on the classic platforms, a traffic spike becomes a fleet of environments — which your downstream database will meet all at once. Compute is disposable; state is the hard part. That sentence will follow us for the rest of the term.

06What Compute Costs

Everything above is machinery for the argument Notes 01 made with sliders: utilization is the variable, and you now control it. The pricing menu rewards you for telling the provider how much idleness you can promise away. The same vCPU is sold three ways. On-demand: no commitment, highest rate — the price of maximum flexibility. Reserved / committed use (Savings Plans, committed-use discounts, reservations): promise one or three years and pay 30–60% less — the right price for load that never goes away. Spot / preemptible: the provider's spare capacity at 60–90% off, reclaimable on minutes' notice — the right price for work that can be interrupted, and a natural partner for autoscaling groups, which simply replace reclaimed instances the way they replace failed ones.

Put next to Plate 5.1 from Notes 01, this menu produces the standard shape of a well-priced fleet: commit to the floor, autoscale the swell, spot the interruptible, and let functions catch the spikes. The baseline that runs at 3 a.m. is reserved; the daily swell rides on-demand or spot behind the autoscaler; the rare, bursty edges — webhooks, batch triggers — go to functions. Each layer is priced for the utilization it actually achieves. A bill that is all on-demand is a sign nobody looked; a bill that is all reserved is the 2004 fleet wearing a cloud costume.

Which leaves the question every project in this course must answer at least once: for this service, at this traffic, are functions or instances cheaper? It is a utilization question, so it has a numerical answer. Set your workload's shape below and find the crossover.

Read the result honestly, in both directions. Below the crossover, functions win twice — they cost less and the instances they replace would mostly have idled, patched and paged, on your time. Above it, instances win on the meter, but the gap is buying back an on-call rotation, OS patching, and capacity planning; sometimes that is a fine purchase and sometimes it is not, and your project write-ups are expected to say which and why. The lab also omits real line items on both sides — free tiers (a million invocations a month, forever, on the major platforms — often the entire bill of a course project), the API gateway in front of functions, the load balancer in front of instances — and duration and memory move the amber line's slope, which is why a 200 ms function and a 2-second function live in different economic universes. Rates are for shape, not memorization: in your Project 1 estimate, cite the current pricing page and state the region, as with Notes 01.

07Before Session 5 — September 21

Project 1 work begins now — the specification you read last week stops being reading material this week. Nothing below is long, but three items produce artifacts your project is graded on, so do them deliberately. The list remembers your progress on this device.

• Review Notes 03 with the lab open. Run the three experiments in Section 04 until you can manufacture thrashing on demand, cure it, and explain in one sentence each what target, cooldown, and warm-up control. In Session 5's opening round you may be asked to drive.
• Provision your Project 1 instance. Smallest current-generation general-purpose type that runs your service — decode the name you chose and write one sentence in your repository README justifying family, generation, and size. “It was the default” is a sentence, but not a justification.
• Make the first real commits. Repository with a README, an architecture sketch (region, zones, where a load balancer would sit if traffic grew), and whatever exists so far. Commit as you work — per Notes 01, your history is evidence of individual work, and a single heroic Friday push reads exactly like what it is.
• Draft the cost estimate at three traffic levels using Section 06's method: name the levels, price the instance path and the function path at each, cite the provider's current pricing page, and state the region. If the cheapest architecture changes between your low and high levels, say so — that observation is the point of the exercise.
• End every work session with a teardown sweep. List running resources and reconcile against what you meant to leave up. The orphaned-resource warning from Notes 01 §5.3 stops being hypothetical the week you start leaving things running — which is this week.

08Key Terms

Instance family

A resource ratio: general purpose, compute-, memory-, storage-optimized, or accelerated. Declares which resource you expect to run out of first.

Instance type / size

Family + generation + size, e.g. m7g.xlarge. Within a family, vCPU, memory, and price scale linearly with size.

Burstable instance

Cheap types (t, e2, B-series) that bank CPU credits while idle and throttle when the credits run out. For idle-mostly workloads only.

Machine image

The template a boot volume is created from (AMI / custom image / managed image): OS plus whatever you baked in.

Golden image

An image with everything pre-installed, so boot-to-useful is seconds. Maintained as a built, versioned artifact.

User data / cloud-init

First-boot configuration: a script the instance runs on launch. The configure-on-boot end of the image spectrum.

Immutable infrastructure

Never patch running instances; build a new image, roll new instances, destroy the old. Container-image philosophy at VM granularity.

Load balancer

One stable address that spreads requests across registered instances, making N interchangeable machines look like one. Depth in Week 5.

Health check

A periodic probe of each instance. Failures pull it from rotation (load balancer) and get it replaced (autoscaling group).

Autoscaling group

Launch template + min/desired/max + policies (ASG / managed instance group / VM scale set). Makes instance count an output of policy.

Target tracking

A thermostat policy: hold a named metric at a setpoint by adding and removing instances. The sane default.

Cooldown

Minimum quiet period between scaling actions — the hysteresis that prevents thrashing.

Thrashing / flapping

Rapid oscillating scale-out and scale-in driven by metric noise, a tight band, or no cooldown. Costs money and stability.

Cold start

The latency of creating a fresh function execution environment — boot the microVM, load runtime and code, run init. ~100 ms to seconds.

Scale to zero

Running — and paying — nothing when there is no load. Functions do it natively; instance fleets bottom out at their minimum.

Spot / preemptible

Spare capacity at 60–90% off that the provider can reclaim on minutes' notice. For interruption-tolerant work, ideally behind an autoscaler.

No term matches that filter.