LoRA on spot RTX 4090s: what happens when the host dies mid-run.

Community GPU marketplaces like Vast.ai rent an RTX 4090 for well under a dollar an hour. For a lot of fine-tuning
work that is the cheapest compute you can get. The catch is that the machine belongs to someone else: hosts reboot,
lose their network, get outbid, or simply disappear in the middle of your run.

I wanted to know two things: how much cheaper a real fine-tune is on such a machine, and what it takes for that
fine-tune to survive the host disappearing.

The job

• QLoRA fine-tune of Qwen2.5-7B-Instruct (4-bit), yahma/alpaca-cleaned, 300 steps, checkpoint every 50 steps.
• Public image pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime, dependencies installed at start.
• One RTX 4090, host reliability ≥ 0.98, network ≥ 500 Mbit/s.

What it cost

The number below is taken from the Vast.ai invoice for one run without failures, not from "price × time".

So 86–89% cheaper on this job. Not "90% on everything": other jobs, idle time and moves between machines change
the number. The summary is on the landing page.

Six later runs of the same job on 4 November, on six different hosts and without failures, were billed $0.21–0.38 by
Vast. The most expensive one ran on a host that trained at about 6 s per step instead of about 4, so training took
31 minutes instead of 17–19.

One thing that surprised me: on the first attempt the bill was $0.81, not $0.19. The GPU hour was cheap, but the host
charged for traffic, and downloading a 15 GB model ate most of the savings. Since then the host is chosen by the full
hourly price including traffic, not by the GPU price alone.

Killing the machine on purpose

Then the interesting part. Halfway through training I destroyed the instance by hand.

Watch the recording, about 60 s: training → instance destroyed → new machine → "restored from checkpoint" → training continues.

What happened:

1. Within about half a minute the supervisor noticed the agent had stopped reporting and marked the host as lost.
2. It rented a replacement with the same requirements.
3. The new machine downloaded the image and the model (this is where most of the ~5 minutes of downtime went).
4. Training continued from checkpoint 100, not from step 0. Nothing to do on my side; I got two Telegram messages — "host lost" and "running again".

In a separate test I destroyed the machine twice during training: the job continued after 3.5 and 13 minutes.

How it works

• An agent in the container reports a heartbeat every minute and uploads checkpoints to object storage (Cloudflare R2) every few minutes, as soon as the training script marks one ready.
• A supervisor outside the GPU machine watches all jobs. No heartbeat, host gone, GPU not responding, or no progress for too long — it rents another machine and the job resumes from the last checkpoint.
• Hard spending limits per job, day, week and month: warning at 80%, at 100% the machine is released and progress is kept.
• Each job gets a temporary storage key that only works for its own folder; checkpoints can be encrypted inside the container before upload.

Live numbers, not just one demo

One staged failure proves little, so the supervisor publishes its own numbers on a
public status page. On 9 November — on our own test jobs, so a small sample —
it showed 15 job interruptions over 30 days and 12 jobs resumed, with a median of about one minute and a 90th percentile
of eight minutes until the job was back on a machine. The other three were my own test jobs, cancelled before the
replacement came up.

What it does not solve

• The host can see your data while the job runs. That is true of every GPU marketplace. Don't run personal or confidential data this way; a data-center-only mode reduces the risk but does not remove it.
• Start-up is slower than a big cloud: every new machine downloads the image and the model. In six runs on 4 November the first training step came 4.5–8.5 minutes after renting, and those minutes are billed.
• One GPU per job for now — no multi-node training.
• Your code is still yours: if the script crashes because of a bug, moving to another machine won't fix it.

Try it

I packaged this as NomadGPU: it runs on your own Vast.ai account (you pay Vast directly), and the first 14 days are
free. If you have a fine-tune that keeps dying with its host, I'm happy to set it up with you:
nomadgpu.com.

Questions about the method or the numbers — ask below, I'll answer with the logs.