Cold start is not a myth. It is also not a single number. When we built ServerlessFlow — a self-hosted FaaS on OpenWhisk and Docker — the useful move was to stop arguing about serverless and start timing the three phases of a start.

WARM POOL ready ready busy spare hit invoke <200ms reuse path COLD PATH allocate → init → load · pay this before the user waits
Pool hit rate is the product metric. Spawn time is the cost metric.

Three clocks, not one

A "cold" invocation is roughly:

  1. Allocate — create or schedule a container
  2. Init — runtime, mounts, env, your process
  3. Load — import the function, connect to anything you foolishly connect to at import time

If you measure only wall time from HTTP to handler, you will optimize the wrong phase. We watched spawn time and pool hit rate as first-class metrics. A beautiful handler in a container that took a second to exist is still a slow product.

On a node with a warm pool, reuse is the common case: the process is up, the runtime is hot, you dispatch the payload. That path is what people mean when they say "feels like a server." The cold path is what they mean when they say "serverless is unusable."

Both are the same system. The difference is whether you already paid for init.

Pools are a forecast, not a vibe

A static pool of 50 containers is easy and wasteful. A pool of 0 is honest and slow.

We sized from recent concurrency: if we have been running 40 functions, keep a buffer above that. If we have been idle, shrink. Predictive scaling based on a 15-minute velocity beat both "always warm" and "always cold" for our traffic, which is spiky but not random.

The scheduler's only interesting rule: prefer an idle warm container over a create. Creates happen at the edge of the pool, not on the first request of a working day.

Pre-warming is not magic. It is running init before the user is waiting. You still pay CPU and memory. You just pay it in the background, where product people cannot see a spinner.

What we actually optimized in the function

After allocation was in the hundreds of milliseconds instead of seconds, load time started to matter:

  • Stop connecting to databases at import time
  • Stop shipping a 40MB node_modules into every image "just in case"
  • Prefer a thin runtime image per language over a kitchen-sink base

Tree-shaking a Node function helps. It does not help as much as not booting Docker. Sequence your work: pool first, bundle second, micro-optimizations never if you cannot see them on a graph.

Isolation is why you are not using a shared Node process

A warm pool is one step from "we have a fleet of long-lived workers." The reason to stay in function-shaped boxes is blast radius: cgroups, one user's memory leak, one infinite loop.

If you do not isolate, you do not have FaaS. You have a monolith with extra YAML. If you isolate but never reuse, you have isolation with a latency tax.

Your numbers will differ. Your region, image, and import graph will differ. Copy the method: measure allocate vs init vs load, put a pool in front of allocate, and make hit rate visible to whoever owns the pager.

If a vendor's cold start bothers you, you can buy provisioned concurrency — or you can run a pool yourself and own the bill. Both are valid. Pretending start time is a moral failing of the cloud is not.