A serverless GPU is one you never provision: the platform starts your container when work arrives, bills while it runs, and scales to zero when traffic stops. Nodus Functions work this way, starting workers when calls arrive and scaling them back down, while long training runs go to Nodus TrainingJobs, which checkpoint and resume instead of hitting a call timeout. Use serverless for bursty inference, short batch jobs and agent tool calls, and an always-on GPU or a training job once the GPU would be busy most of the day.

How does a serverless GPU actually work?

You hand the platform a container image and a function, and it handles the machines. When a call comes in and no worker is free, it places a worker on a GPU, pulls the image, loads your code and runs the call. That wait is the cold start.

After the call, the worker stays warm for a short idle window in case another call arrives. If none does, it is released and you stop paying.

On Nodus that window is scaledown_window, one minute by default and up to 20 minutes, as the Functions scaling guide describes. Modal keeps containers alive for 60 seconds after the last input by default and bills for that time, according to its pricing page. Runpod Serverless waits 5 seconds by default before scaling a worker down, per its serverless pricing docs.

Four settings decide most of the behavior on any platform:

Setting What it controls Nodus Functions default
Minimum workers Workers kept warm with no traffic (0 means scale to zero) min_workers 0
Maximum workers The ceiling on parallel GPUs max_workers 10 (1 to 1,000)
Idle window How long a finished worker waits before release scaledown_window 1m
Call timeout The longest one call may run 5 minutes, 24 hours at most

Defaults from the Nodus Functions guide and scaling guide, checked November 17, 2026.

What exactly do you pay for?

You pay for worker time, not for requests, and worker time includes more than your code running. Runpod's docs list three billed phases: the cold start until your handler is ready, execution, and the idle wait after the last job. Runpod bills from worker start to full stop, rounded up to the nearest second, and does not bill image pulls that happen while the worker is scaled down (Runpod serverless pricing). Modal bills during application load time and while inputs are processed, per second (Modal pricing).

On Nodus, the estimate tells you the rate and the start times before you deploy. Calling embed.estimate("hello") returns the expected cost, the cold and warm start bands and the worker's hourly rate, and every finished call reports status.costUSD, the worker time attributed to it (Calling Functions).

So a serverless bill is busy time plus overhead. If each call loads a large model from scratch, the overhead can exceed the work.

What is the difference between a request and a job on a serverless GPU?

A request is a call someone is waiting on, and a job is work you start and collect later. On Nodus, .remote() blocks until the result comes back, .spawn() starts a call and returns a handle, and .map() fans one Function out over many inputs and returns results in input order (Functions guide).

The difference matters for timeouts. Synchronous requests should finish in seconds, and jobs can run longer, but every platform caps a single call.

A Nodus Function call that runs past its timeout ends as Failed with DeadlineExceeded. Modal's default execution timeout is 300 seconds and can be raised to 24 hours (Modal timeouts). Runpod's console default is 600 seconds, with a range of 5 seconds to 7 days (Runpod endpoint settings).

Why are cold starts the real limit for inference?

Cold starts, not price, are what make serverless awkward for latency-sensitive endpoints. Most of a GPU cold start is loading model weights into GPU memory, and that grows with the model.

Nodus documents one measurement on an NVIDIA A10. A vLLM engine serving Qwen3-0.6B took about 90 seconds to cold start, and about 6 seconds to restore from a GPU snapshot with its weights and KV cache kept out (GPU snapshots (Beta)). Snapshots are in Beta. If your users cannot wait a minute and a half on the first request after a quiet period, you either keep a worker warm with min_workers=1 or make that restore path part of the design.

Keeping one worker warm trades latency for an always-on bill. The Nodus scaling guide shows the pattern:

@app.function(cpu=2, memory="4Gi", min_workers=1, max_workers=20, scaledown_window="5m", target_concurrency=4) 

That keeps one worker up, adds one worker for every four calls waiting or running, and holds idle extras for five minutes.

When does serverless cost less than an always-on GPU?

Serverless wins when the GPU would sit idle most of the day, and loses once it is busy more than the ratio of the two prices. That gives a rule you can check against any price list.

Break-even busy fraction = always-on hourly price / (serverless hourly price x (1 + overhead))

Overhead is the billed cold start and idle time on top of real work, as a fraction of busy time. We derived this rule from each vendor's published billing model; it ignores storage and egress.

Runpod makes a clean worked example because it sells both models for the H100. Its pricing page lists an H100 PRO serverless flex worker at $5.89/hr and an H100 SXM Pod on Secure Cloud at $4.39/hr.

  • Bursty, 4 busy hours a day: serverless costs 4 x $5.89 = $23.56. An always-on Pod costs 24 x $4.39 = $105.36.
  • Steady, 20 busy hours a day: serverless costs 20 x $5.89 = $117.80, more than the Pod's $105.36.
  • Break-even with zero overhead: $4.39 / $5.89 = 74.5% busy, or about 17.9 hours a day.

On Runpod, Serverless flex costs $23.56 a day at 4 busy hours and $117.80 at 20 busy hours, while an always-on H100 Pod costs $105.36 a day.
Under about 18 busy hours a day, paying only for busy time is cheaper even at a higher hourly rate.

Overhead pulls the break-even down. If cold starts and idle tails add 25% to busy time, break-even falls to 74.5% / 1.25 = 59.6%, about 14.3 hours a day. At 50% overhead it is 74.5% / 1.5 = 49.7%, about 11.9 hours. These overhead levels are illustrations, not measurements; read your own from the platform's per-call cost or usage records.

Break-even busy hours per day for Runpod Serverless versus an always-on Pod: 17.9 hours at 0% overhead, 14.3 hours at 25% and 11.9 hours at 50%.
Slow cold starts make serverless stop paying off sooner, so shrink load time before you compare prices.

The rule changes shape when a platform charges the same rate for warm and busy workers. Modal lists an NVIDIA H100 at $0.001097 per second, which is about $3.95/hr, whether a container is serving or held warm (Modal pricing). On Nodus, an idle warm worker is billed at the Function's worker rate, the same as a busy one, and the H100 SXM is listed at $2.60/hr on the pricing page. When warm and busy time cost the same, scale to zero never costs more for the same busy time; the trade is cold-start latency against the cost of min_workers.

Option H100 list price How idle time is billed
Nodus $2.60/hr Warm idle window billed, released after scaledown_window
Modal $3.95/hr ($0.001097/s) Idle window billed, 60 s by default
Runpod Pod, Secure Cloud $4.39/hr Always billed while running
Runpod Serverless flex $5.89/hr Idle timeout billed, 5 s by default

Sources: Nodus pricing, Modal pricing, Runpod pricing, Runpod serverless docs. Prices checked November 17, 2026.

H100 list prices per hour: Nodus $2.60, Modal $3.95, Runpod Pod $4.39 and Runpod Serverless flex $5.89.
The serverless premium varies by platform, and on some it is zero.

For a ranked look at platforms, see the best serverless GPU platforms in 2026.

When is serverless the wrong choice?

Serverless is the wrong tool for work that keeps a GPU busy for hours. Steady high-utilization serving crosses the break-even above, and long training runs into the call timeout. A 30-hour fine-tune does not fit in a 24-hour call, and a call that fails partway has no built-in progress to resume.

On Nodus that work belongs in a Job or a TrainingJob. A TrainingJob runs a managed runtime such as nodus/sft or nodus/grpo-lora, prices itself in a dry run, takes a maxCostUSD cap, and recovers from full checkpoints with optimizer, scheduler and RNG state (Training (Beta)). We cover the long-run case in detail in can you use serverless GPUs for long training runs.

A middle case is worth knowing. A Nodus GPU class that scales to zero leaves its GPU idle between workers, and with gap-fill (Beta) a checkpointed one-GPU Job from your organization can use that idle time. On an A10, such a Job kept 91% of a dedicated GPU's steps at one request a minute and 34% at eight, per the GPU snapshots guide.

Which Nodus feature should you pick?

Pick by how long the work runs and whether someone waits on it. The compute feature guide maps the full set; for GPU work it comes down to this:

  • Calls in seconds to minutes, bursty or fanned out over many inputs: Functions, with min_workers=0 unless latency demands a warm worker.
  • A script or batch that should run to completion: a Job.
  • Fine-tuning, preference training, distillation or RL over hours or days: a TrainingJob.
  • Calling a hosted open-weight model rather than serving your own: Inference (Beta).

To start, install the SDK and sign in, as the docs show:

$ pip install nodus-compute $ nodus login 

FAQ

Is a serverless GPU the same as a spot GPU?

No. Serverless describes how you are billed and scaled, per unit of worker time with scale to zero. Spot describes capacity that can be reclaimed at a discount. A platform can combine them, so check both before you plan a long run.

Does scale to zero mean my model reloads on every request?

Only on requests that arrive after the idle window has closed. Calls inside the window reuse the warm worker, which is why the idle window is the cheapest latency control you have.

How do I measure my real overhead?

Compare billed worker time with time spent inside your handler over a day of real traffic. On Nodus, each call's status.costUSD and the usage records give the billed side.

Can agents use serverless GPUs?

Yes, and agent tool calls are a good fit because they are short and irregular. A coding agent such as Claude Code can also start Nodus work through MCP after nodus mcp install, following the connect guide.

Originally published on the Nodus blog.