Cold Start Latency on Serverless Inference: What Actually Happens in Those Few Seconds

A cold start is five waits, and they don’t share a cause.

• Phase 1: scheduling and capacity allocation. Before anything else, the platform finds or spins up a GPU with enough free memory for your model. How long this takes depends on how full the cluster is and how fast the scheduler can place your workload, not on your model at all. This is the phase users never see directly, the one that varies most run to run, and the one a self-hosted reproduction cannot observe. On your own GPU Droplet, there’s no multi-tenant scheduler deciding whether a GPU is free. Any white-box experiment is blind to this phase by construction. Track B below is the only one of our two measurements that can see it, and even then only indirectly.

• Phase 2: container and runtime start. The platform pulls the container image (if it isn’t already on that node), starts the container, and initializes dependencies. This is the phase generic serverless discourse fixates on, and it’s often the smallest contributor once a platform has decent image caching. Our measurements bear this out, with a caveat that surprised us and that we discuss below.

• Phase 3: weight transfer into VRAM. The model’s weights move from object storage or local NVMe into GPU memory. Duration is roughly weight bytes divided by effective bandwidth, so it scales with model size and depends heavily on where the weights are staged. This is the phase quantization changes directly, and it’s the phase most people assume dominates. Our data says it only dominates when the cache is cold.

• Phase 4: serving runtime initialization. CUDA context creation, engine startup, KV cache allocation, and, distinctly, CUDA graph capture the first time the engine sees a given batch shape. DigitalOcean’s own Ornith-9B benchmark already showed this phase is real and separable: a first request at a new concurrency level hit 0.691s time-to-first-token (TTFT) versus roughly 0.035s on every subsequent request, on already-warm hardware. That is a ~20x gap from CUDA graph capture alone. Our decomposition shows this phase is the dominant cost of a warm-cache cold start.

• Phase 5: first-request prefill. The actual first inference, turning your prompt’s tokens into the model’s internal representation before generation starts. This looks like ordinary TTFT, but on a cold start it stacks on top of everything above. Providers sometimes quote a “cold start” number that’s really cold-start-inclusive TTFT, and sometimes one that excludes prefill entirely. Knowing which one you’re looking at is the difference between a useful number and a marketing number.

We ran two experiments, and keeping them separate matters.

• Track A is the white-box decomposition. We deployed vllm/vllm-openai:v0.26.0 from scratch on a single H100 80GB GPU Droplet and timestamped every phase directly, from container start through first prefill, by hooking vLLM’s own startup log lines. We ran this across four Qwen3 model sizes (0.6B, 1.7B, 4B, 8B) and five cache conditions, from everything cold to everything warm, for 200 trials total, ten per cell. Track A can decompose what a black-box test only sees in aggregate. Its one hard limit, stated up front: it approximates phases 2 through 5 but cannot observe Phase 1, because there is no multi-tenant scheduler on a Droplet you own.

• Track B is the black-box observation. Here the plan changed on contact with reality, which is why it gets its own section below. We measured what a user of DigitalOcean’s serverless inference platform actually experiences, probing every five minutes for seven days.

Three limits on scope are worth naming up front. We measured Track A across 0.6B to 8B, not a full 70B phase breakdown: the 70B appears only as a black-box control in Track B. We did not run a quantization/precision sweep, so the Phase 3 delta from 4-bit weights is reasoned from byte counts, not measured here. And the per-trial harness and data are published separately (see the repository linked in the note above) rather than reproduced inline. The scaling trends below are strong enough across four sizes to extrapolate the shape to 70B, but the 70B phase split itself remains predicted, not measured.

The five cache conditions in Track A, each leaving one more thing warm than the last, are what let us attribute time to specific phases:

We adopted DigitalOcean’s own TTFT definitions and followed the metrics-that-matter protocol: multiple trials, warm-ups discarded, sampled across time-of-day windows, so the numbers are comparable to other work in this series.

The original plan for Track B was the standard one: let a serverless deployment sit idle past its scale-to-zero window, send a request, and measure how much slower it is than a warm one. This is the methodology every cold-start article reaches for. It does not work on pooled serverless inference, and understanding why reframes the entire problem.

DigitalOcean Serverless Inference pools GPU capacity across all customers. There is no per-customer instance sitting idle, no deployment ID, no scale-to-zero window that belongs to you. Your silence does not make the model cold; other tenants’ traffic keeps it warm. The function you’d need to write, “wait until my deployment has scaled to zero,” is not merely hard to implement; it is meaningless, because there is no my deployment to scale.

We found this out the direct way. A first pass using the forcing design produced 41 samples with a median TTFT of 1.05 seconds and no relationship whatsoever between idle duration and latency. Cold requests were faster than warm ones in 10 of 41 trials. That’s not a failed measurement. It’s a correct measurement of a pool that was warm every single time we probed it.

So we switched from forcing a cold start to observing for one. Cold starts still happen on a pooled platform, because capacity scales with aggregate demand and models get evicted, but you cannot cause one. You can only sample continuously and catch one when it happens. The consequences for methodology:

• Probe frequently, for a long time. We sampled every five minutes for seven days (1,827 usable probes) instead of a few dozen forced trials.
• The tail is the story, not the median. On a warm pool the median is just “the pool was warm,” which it almost always is. The question is how often a user hits something slow, and how slow.
• Model popularity replaces idle duration as the independent variable. A flagship model is kept warm by everyone else’s traffic; an obscure catalog model is where a scale-up is most likely to be observable. We probed a low-traffic target (alibaba-qwen3-32b) against a popular control (llama3.3-70b-instruct) in the same sample, so a spike on one but not the other tells us whether the cause was model-specific or platform-wide.

The control probe is what makes the Track B numbers trustworthy rather than anecdotal. A latency spike seen from outside the pool could be a model load, network variance, congestion, or client-side connection setup. Probing a known-warm control alongside the target, timing the network path separately, checking whether inter-token latency stayed normal, and immediately re-sending each request gives four independent ways to tell those apart.

The design rests on an assumption worth stating: cold starts on a pooled platform track model popularity, because popular models justify warm replicas and niche ones don’t. That isn’t ours to claim alone. DigitalOcean’s consistency benchmarking reaches the same three conclusions we built Track B on, from independent testing: measure the coefficient of variation rather than the median, because cold starts produce a bimodal distribution the median hides; sample at off-peak hours, because cold-start behavior is worst when traffic to a model is lowest; and expect low-traffic models to cold-start more often than warm flagships. Our obscure-target-against-popular-control design is that reasoning applied to a single platform over time.

Here is the Track A decomposition for the 8B model, from fully cold (C0) to fully warm (C4), measured as median seconds from container start to first token:

First, the naive cold start is 157 seconds, of which about 63 seconds (the 8B C0 image-pull median, a Phase 2 sub-measurement not shown in the totals table above) are the container image pull, which the folklore says is negligible. It’s negligible only if the node already has your image; on a truly cold node it’s the single largest line item. Second, note that C1 is faster than C2. When the weights are deleted (C1), vLLM downloads them during the trial and they land warm in the page cache; when they’re present on disk but the page cache is dropped (C2), the engine pays a genuinely cold read. Downloading weights and reading them from a cold disk cost about the same. That inversion is the first hint that the page cache, not the weight source, is what matters.

Figure 1. Phase decomposition for Qwen3-8B across cache conditions. The C0 total is dominated by the 63s image pull; C4 bottoms out at 53s.

Now the floor. Once every cache is warm (C4), the 8B cold start bottoms out at 52.9 seconds. Here are the largest components vLLM’s own logs let us time directly:

These are the log-marked phases, not a complete partition: they sum to ~43.8 seconds, and the remaining ~9 seconds is container start plus the unmarked gaps between phases. The shape is what matters. Only the 8 seconds spent loading weights scale with model size. The rest is fixed process, CUDA, and graph-capture cost that is identical whether you’re loading a 0.6B model or an 8B one. The proof is in how little the floor moves across a 13x range of parameters: 48.6 seconds at 0.6B versus 52.9 seconds at 8B. A model with thirteen times the parameters adds about four seconds to the warm-cache floor.

Figure 2. The same decomposition across all four sizes. The C4 floor rises from 49s to 53s across a 13× parameter range.

The compile cache is a separate lever from the page cache, and it acts on a different phase.

Figure 3. torch.compile time by condition. The compile cache cuts it ~4× (C2→C3); warming the page cache (C3→C4) barely moves it.

Here the data contradicts the intuitive anatomy. Phase 3, weight transfer, the phase everyone assumes dominates, shrinks the most as you warm the OS page cache (the compile cache cuts Phase 4 separately), and at the floor it is smaller than either CUDA init or graph capture. The bottleneck is Phase 4: CUDA context creation and graph capture, which no amount of weight pre-staging touches. Caching is necessary but not sufficient. You can eliminate the image pull, pre-stage every weight, and warm the page cache, and you will still wait ~50 seconds, because that time is the engine starting up, not the model loading.

If Phase 3 (weight load) is small at the floor, it’s enormous when cold, and the thing that flips it is the OS page cache, not where the weights are staged.

We measured weight-load time against model size under each condition. With a cold page cache, weight loading scales at about 1.42 seconds per GiB. With a warm page cache, the marginal scaling drops to about 0.15 seconds per GiB on top of a fixed ~5–6 seconds load overhead, a roughly 90% reduction in the size-dependent part. The per-model deltas:

Figure 4. Weight-load time vs model size. Cold page cache scales at ~1.5 s/GiB; warm collapses to ~0.15 s/GiB.

For the 8B model, whether the weights happen to be in the OS page cache is worth 18 seconds, larger than the compile-cache delta (~13 seconds, the phase-specific torch.compile median delta for the 8B model) and larger than any other single avoidable cost we measured. And unlike the fixed floor, this one scales with model size, so it only gets worse for the 70B-class models most people actually worry about.

This reframes the “pre-stage weights on local NVMe” advice. Staging weights on fast local disk helps, but the mechanism that actually pays off is the page cache: the second read of a file is fast because it’s served from RAM, not disk. On a serverless platform, that’s the difference between a request landing on a node that recently served your model and one that didn’t, which is the kind of locality a pooling scheduler is trying to exploit.

Now the black-box view. Seven days at five-minute intervals produced 1,829 legitimate probes. Of the legitimate probes, 1,827 produced a usable target-model TTFT; the other two were one 500 and one 200 with no tokens, both recorded as failures rather than counted as fast. So the usable rate is 99.9%, and the percentages below are over legitimate target-model probes. One caveat to note through this whole section: these are one week, from one client region, against one target and control pair. The numbers are specific to that setup, not a universal characterization of the platform.

Figure 5. Target-model TTFT tail (log scale). 99% of probes finish under 3.9s; the slowest was 9.3s.

Set that against the self-hosted floor of 52.9 seconds. The pooled platform served 99% of legitimate target-model probes in under 3.9 seconds, one to two orders of magnitude faster than what it costs to cold-start the same class of model yourself. Pooling works: by keeping models warm across aggregate tenant traffic, it absorbs almost the entire cold start that a self-hoster pays in full.

Of the 1,827 usable samples, 11 (0.60%) survived every discriminator as genuine cold-start candidates: target slow, control fast, inter-token latency normal, immediate retry fast. Their median was 3.9 seconds and their max was 9.3 seconds, clustered in low-traffic UTC morning hours, consistent with the pool having scaled an unpopular model down. So cold starts do happen, and when they do they’re consistent with a quick scale-up, not a 53-second from-scratch load. Even the slowest target-model cold-start candidate (9.3 s) was faster than the self-hosted floor. That is a separate matter from the control-model incident discussed next, which was far larger.

The tail has a second story that should shape the architecture decision. The control model, llama3.3-70b-instruct, the popular control expected to stay warm, had a p99 of 41 seconds and a single worst sample of 287 seconds. Almost all of that came from one day:

On six of seven days, the popular model’s tail was a benign ~1.5% of samples over 10 seconds, the pool’s ordinary variance. On the seventh, that jumped roughly fivefold, with first-token times reaching 287 seconds, while the target model stayed fast. That divergence, one model degrading badly while another on the same platform is unaffected, is why the control probe earns its place: it tells us this was a model-specific platform incident, not a network problem or a measurement artifact. The day was fully sampled (288 of 288 slots, zero failures), so the 7.8% is exact, not an undercount.

Figure 6. 15 Aug, control vs target. Control (llama3.3-70b) spikes to 287s while the target stays flat — a model-specific incident, not a platform-wide one.

Pairing the two tracks: a pooled platform absorbs the routine cold start extremely well, but you inherit the platform’s incident tail, and that tail can briefly exceed the fully cold self-hosted baseline we measured for this 8B setup. Self-hosting this 8B setup gave a bad but repeatable fully cold baseline: a ~157-second median, consistent across our runs, under your control. Pooled serverless gave a much better median and p99, plus a rare tail event you don’t control and can’t predict. Which risk profile is right depends entirely on your workload, which is the next section.

Every mitigation attacks one specific phase. That mapping is what makes a provider’s “fast cold start” claim checkable instead of a slogan.

• Phase 1 (scheduling): warm pools and minimum-instance floors. Keep capacity pre-allocated so a request never waits on the scheduler. On a pooled platform this is what the provider is doing for you invisibly, and our Track B data shows it working for 99% of legitimate target-model probes. The residual risk is the incident tail. Keep-alive traffic can’t fix it because you don’t control the pool, though retries, model or provider fallback, and degraded-mode routing can reduce your exposure to it.
• Phase 2 (container start): node-local image caching. Our data reframes this one. The image pull is negligible only when the node already has the image, but when it doesn’t, it’s 63 seconds, the largest single cost in a fully cold start. On a platform with good node-local caching this is close to solved; on a cold node it dominates. Slimmer images help the cold-node case directly.
• Phase 3 (weight transfer): warm the page cache, then worry about the source. Quantization still helps (fewer bytes to move), but our numbers say the bigger lever is page-cache locality: 1.42 s/GiB cold versus 0.15 s/GiB warm. Pre-staging weights on local NVMe matters mostly because it makes that warm second read possible. For a 70B-class model this is tens of seconds of swing.
• Phase 4 (engine init): pre-capture CUDA graphs and keep the context alive. The floor measurement moves this phase from footnote to headline. Roughly 25 of the 53 warm-floor seconds are CUDA init plus graph capture, and they’re fixed cost, proportionally worse for small models. Pre-capturing graphs for expected batch shapes and holding a persistent CUDA context across requests is the only thing that touches this phase. If a provider’s “fast cold start” story is entirely about weights, it hasn’t addressed the phase that actually dominates a warm-cache start.
• Application-level, phase-agnostic: On serverless systems with a customer-controlled deployment and scale-to-zero window, keep-alive pinging can prevent scale-to-zero, but it pays for warmth per request instead of per hour—the same tradeoff a reserved-capacity decision makes, structured differently. Enough keep-alive traffic and pre-warming schedules add up to a system paying for dedicated capacity without admitting it. Name that honestly rather than treating serverless-plus-workarounds as free.

One disambiguation worth stating plainly, since it’s a common confusion: prompt caching does not reduce cold starts. Prompt caching addresses repeated input content on an already-running deployment; a cold start is the deployment itself reloading after scaling to zero. Different mechanisms, different problems. A workload with both needs both solutions.

Whether serverless is cheaper than dedicated capacity for your traffic is a separate question from whether your workload can tolerate the cold starts serverless comes with. Both have to hold. Our data lets you reason about the second one concretely, using the tail, not the median.

• Tolerate cold starts, no mitigation, when the workload is async or batch (embeddings, offline scoring, anything without a human waiting), it’s an internal tool where occasional delay isn’t a product problem, or (per Track B) you’re on a pooled platform whose measured TTFT distribution fits your SLA, where the routine tail is low (our target model held a sub-4-second p99) and you can absorb a rare multi-minute incident like the one the control model hit.

• Serverless plus mitigation, when the workload is user-facing but bursty enough that a minimum-instance floor covers the traffic cycle without paying for capacity around the clock. The math is a direct comparison: one warm floor instance against full dedicated capacity, weighed against the operational cost of managing a warm pool yourself.

• Dedicated capacity, when cold-start-inclusive p99 violates your SLA outright, or (the case Track B adds) when you cannot tolerate a rare uncontrolled tail event even if the median is excellent. Self-hosting’s ~157-second fully cold median (measured for this 8B setup) is bad, but it’s yours: repeatable, predictable, and something you can eliminate with a warm pool you control. The pooled platform’s 287-second incident is rare and someone else’s to fix, which is either fine or unacceptable depending on what’s waiting on the response.

The latency decision has an economic gate underneath it. DigitalOcean’s token-economics analysis works out the crossover for a 70B model: a dedicated H200 GPU Droplet only beats serverless per token above roughly 72% sustained GPU utilization; below that, the idle capacity you pay for around the clock makes serverless cheaper. Cold-start tolerance and that utilization threshold have to point the same way. A bursty, user-facing workload tends to sit below the utilization line and be the least able to absorb cold starts, which is the combination that pushes you toward serverless-plus-a-warm-floor rather than either extreme. The inference trilemma framing of TTFT and inter-token latency (ITL) at p50/p95/p99 is the vocabulary to hold your SLA in while you decide.

A five-question checklist for any provider’s “fast cold start” claim, one per phase: How much capacity is pre-allocated versus scheduled fresh (Phase 1)? Are images cached locally on serving nodes (Phase 2)? Are weights pre-staged near the GPU and is the page cache warm (Phase 3)? Is the serving engine’s graph and context pre-warmed for your batch shapes, or compiled fresh on first use (Phase 4)? And is the “cold start” number inclusive of first-request prefill, or measured separately (Phase 5)? A provider who can answer all five with specifics is telling you something real. One who answers with a single aggregate number is telling you what every competitor’s marketing page says.

When a provider won’t answer, measure it the way Track B did. DigitalOcean’s consistency guide gives a rule of thumb worth stealing: fire at least 75 spaced requests at the model, then compute the coefficient of variation (standard deviation over mean) of TTFT rather than the median. Its bands are a usable gate: under 40% means the model is warm and optimized, over 100% means you’re hitting cold starts, and over 300% means treat it as not production-ready for latency-sensitive work without a dedicated endpoint. Run it at off-peak hours, where the tail is worst.

1. What actually happens during a cold start?

A cold start breaks down into five distinct steps, each introducing its own delay and complexity:

1. GPU scheduling: The platform finds and schedules an available GPU to handle your job.
2. Container startup and image pull: It launches a container and, if needed, pulls the image onto the node. This can be negligible if already cached locally, or substantial if not.
3. Model weight transfer: Model weights are loaded into GPU memory (VRAM). This step’s duration varies based on how large the model is and whether the weights are already in the OS page cache or must be read from storage.
4. Serving engine initialization: The inference engine, often vLLM or similar, initializes. That can include creating the CUDA context, allocating the KV cache, and performing CUDA graph capture. These are mostly fixed costs and become the bottleneck when all caches are warm.
5. First prefill: The system processes the prompt before generation starts, which is the cold-start-inclusive TTFT a user actually feels.

Each phase has different tuning levers and can be impacted by hardware, architecture, and software stack choices. “Cold start” refers to the combination, not any single piece—so end-to-end timing hides the underlying breakdown.

2. Doesn’t loading model weights cause most of the delay?

Only if you’re reading from cold storage. If the OS page cache is warm, loading weights is fast and only a small part of overall latency: roughly 8 of 53 seconds in the warm-cache floor for the 8B model. What dominates there is engine setup: initializing the CUDA context and capturing computational graphs, steps not affected by where the weights reside or how quickly they transfer. Providers and users often optimize model loading because it is visible, but once caches are warm, weight-loading improvements have limited room to move the total.

3. Does using a smaller model meaningfully reduce cold-start times?

At the warm-cache floor, no, not significantly. The floor shows limited sensitivity to model size: 0.6B parameters takes about 48.6s, while 8B takes about 52.9s, a four-second difference despite a more than tenfold increase in size. The reason is that most of the initialization cost is fixed: container launch, CUDA setup, engine startup, and graph capture. Only the model weight transfer step shrinks meaningfully for smaller models, and that phase is not the main bottleneck once caches are warm.

4. What’s the most effective way to reduce load time?

Warm up the OS page cache holding model weights. If the weights must be read from cold disk, loading scales at about 1.42 seconds per GiB. If they are already in RAM through the page cache, the marginal scaling drops to about 0.15 seconds per GiB, on top of a fixed ~5-6 second load overhead. For the 8B model, that page-cache difference saved about 18 seconds compared with a cold page-cache read. Storing weights on fast local NVMe helps mainly because it makes repeated cache hits more likely. Of the avoidable costs measured here, this is the largest model-size-dependent lever.

5. Does serverless inference actually avoid cold starts?

For the target model in this seven-day sample, pooled serverless hid routine cold starts extremely well. The platform served 99% of legitimate target-model probes in under 3.9 seconds, dramatically outpacing the self-hosted cold-start floor. Cold-start candidates were rare: 11 of 1,827 usable samples, or about 0.60%, and the slowest target-model candidate was 9.3 seconds. One important caveat: because you are participating in a pool, you cannot force the system to go cold on demand for your own testing; cold starts must be detected by watching for rare slow tail-latency events over many samples.

6. Is serverless always safer for latency guarantees?

No. Serverless shifts your risk from something predictable and under your control to something rare and outside your control. If you self-host, fully cold starts are consistently slow (a median of about 157 seconds in these tests for the 8B setup), but you can eliminate them with a warm pool you control. In this sample, pooled serverless eliminated almost all of those routine delays for the target model, giving excellent median and p99 latency. However, you inherit the platform’s incident tail: during rare model-specific hiccups, responses can spike unpredictably. In this run, the popular control model reached 286.7 seconds on one day while the target model stayed fast. If your application can absorb the occasional multi-minute outlier, pooled serverless can work well. If not, dedicated capacity or a fallback strategy may be the safer choice.

Those few seconds were never one number. It’s five phases, they scale differently with model size, and they get fixed by different engineering work. The measurement makes the shape concrete: a self-hosted 8B cold start has a median of 157 seconds fully cold and 53 seconds at its irreducible floor. Most of that floor is fixed engine-initialization cost that no weight-loading trick can touch, because the phase everyone optimizes is not the phase that dominates once caches are warm. A pooled serverless platform hides almost all of it, serving 99% of legitimate target-model probes in under 4 seconds, while quietly handing you a rare tail you don’t control: in this seven-day sample, on one day, a popular model took up to 287 seconds to answer.

The practical decision comes down to which tail risk you’d rather own. Self-hosting gave us a repeatable ~157 second median cold start for this 8B configuration, a number you can drive down with a warm pool you manage. Pooled serverless gave a sub-4 second p99 on the model we probed, plus an occasional multi-minute incident you can’t predict or fix. Pick the one your workload can absorb, and use the five-question checklist to hold any provider’s cold-start claim to a specific phase.

• The Hidden Cost of Cold Starts in Serverless AI Workloads, DigitalOcean Community. The three-step conceptual anatomy this article expands into five phases.
• DigitalOcean Serverless Inference: A Deep Dive, DigitalOcean engineering blog. Source for the claim that cold starts are “dominated by weight pre-staging, memory availability, and scheduling speed,” tested directly against our Track A decomposition.
• Fine-tuning the LLM Ornith 9b on a single H200 GPU Droplet, DigitalOcean Community. Source of the independently measured CUDA-graph-capture gap (0.691s vs. ~0.035s TTFT), corroborating our Phase 4 finding.
• How Does Prompt Caching Work and When Does It Actually Cut LLM Costs?, DigitalOcean Community. Source of the TTFT-streaming harness pattern adapted for Track B, and the prompt-caching-versus-cold-start disambiguation.
• LLM Inference Benchmarking: Measure What Matters, DigitalOcean engineering blog. TTFT definitions adopted for measurement consistency.
• Metrics that Matter with Serverless Inference, DigitalOcean Community. Measurement protocol (trial counts, warm-up discards, multi-window sampling) applied to both tracks.
• Why your vLLM p99 latency blows up in production, DigitalOcean Community. Tail-latency framing behind Track B’s focus on p99 and the incident tail over the median.
• Serverless vs. Dedicated vs. Batch Inference, DigitalOcean Community. Mode-selection companion to the decision framework.
• Token Economics Across Traffic Profiles on Dedicated GPUs, DigitalOcean Community. Source of the 72.2% dedicated-vs-serverless utilization crossover for a 70B model on an H200, cited in the decision framework.
• The LLM Inference Trilemma: Throughput, Latency, Cost, DigitalOcean engineering blog. SLO framing (TTFT/ITL at p50/p95/p99) adopted in the decision framework.
• Why Serverless Inference Consistency Varies on the Same Model, DigitalOcean Community. Independent corroboration of Track B’s design (CV over median, off-peak sampling, popularity-driven cold starts) and the source of the CV-threshold benchmarking rule in the checklist.