How do you keep training alive now that Runpod spot Pods are gone?

Runpod has retired its spot Pods, so a Pod you start there today is on-demand and other users cannot displace it. If you want interruptible capacity without babysitting restarts, Nodus runs training on it and continues a reclaimed run on a new machine from its last checkpoint. On Runpod itself, training still stops on a $0 balance, maintenance or a hardware failure, so save checkpoints to a network volume and reload them on start.

Runpod spelled its name RunPod until June 2025, which is why most old guides and search queries use that spelling. We build Nodus, so weigh our view accordingly. Every Runpod fact below links to Runpod's own docs or pricing page.

What happened to Runpod spot Pods?

They no longer exist as a product you can rent. Runpod's Terraform provider migration guide says spot Pods (type = "SPOT", interruptible and bid_per_gpu) "were retired by the platform" and that the APIs stopped accepting bids. It tells you to move those configurations to on-demand Pods.

The REST API v2 migration guide agrees. Its table of removed capabilities lists spot Pods, with the old interruptible, podRentInterruptable and podBidResume fields, and gives the v2 alternative as "None". So any script that bid for interruptible Pods needs a rewrite, not a tweak.

What you get instead is an on-demand Pod. Runpod's Pod pricing docs describe it as pay-as-you-go, with resources "dedicated to your Pod" that "cannot be displaced by other users". Savings plans are deprecated too, so the listed on-demand rate is the normal price.

Can a Runpod Pod still stop in the middle of training?

Yes, for reasons other than a reclaim. Runpod's docs name four, and each one decides where your checkpoint has to be.

Sources: Runpod docs on Pod pricing, maintenance and outages and Pod migration, checked November 14, 2026.

Runpod also lists Community Cloud reliability as "Variable" and Secure Cloud as "High redundancy" in its Choose a Pod guide. The common thread: nothing warns your training process before a crash. Plan for a stop with zero notice, and treat any advance email as a bonus.

Where should checkpoints live on a Runpod Pod?

On a network volume if you can, and never on the container disk. Runpod's storage options page spells out the difference:

• The container disk is lost on stop or restart. Anything outside /workspace lives here.
• The volume disk at /workspace survives stops and restarts but is deleted when the Pod is terminated, and it stays tied to one machine.
• A network volume replaces the volume disk at /workspace, survives Pod deletion and attaches to a new Pod.

Two limits shape the choice. Network volumes must be attached when you create the Pod and cannot be detached later, and they are only available for Pods in Secure Cloud (network volumes). On Community Cloud, keep checkpoints on the volume disk and copy them off the Pod with Cloud Sync to your own object storage. Runpod itself says it is not designed for long-term storage and recommends a 3-2-1 backup habit, and network volumes also have an S3-compatible API for pulling checkpoints out.

The save and resume code is the same as anywhere else: write atomically, load on start. Our guide to resuming training after a spot preemption has the PyTorch version. On Runpod, point its state directory at /workspace and make the Pod's start command run your training script, so a restarted Pod picks up where it stopped. After a stop, start the Pod again from the console or the REST API's start action. If its machine's GPU is taken, migrate it or deploy a new Pod on the same network volume (zero GPUs on restart).

How long does a checkpoint take to write to a network volume?

Seconds for an adapter, minutes for a full fine-tune. Runpod lists standard network volumes at 200 to 400 MB/s, with peaks up to 10 GB/s (network volumes). Checkpoint sizes come from our AWS spot training post. A full mixed-precision fine-tune with Adam saves about 14 bytes per parameter. Rank-16 LoRA adapters on the Llama 3 8B and 70B shapes save about 0.50 GB and 2.49 GB.

The arithmetic is size divided by rate. An 8B full fine-tune saves 8 billion x 14 bytes = 112 GB.

At 200 MB/s that is 112 / 0.2 = 560 seconds, about 9.3 minutes. At 400 MB/s it is 280 seconds, and at the 10 GB/s peak it is 11.2 seconds. The 70B LoRA adapter takes 2.49 / 0.2 = 12.45 seconds at 200 MB/s, and the 8B adapter takes 2.5 seconds.

A full fine-tune checkpoint takes minutes at the listed rate, so it cannot be saved often for free; an adapter takes seconds.

Our decision rule comes from the same budget Nodus uses for automatic intervals: keep checkpointing to about a tenth of run time. That means an interval of at least 10 x save time. For the 8B full fine-tune at 200 MB/s, that is 10 x 560 s = 5,600 s, about 93 minutes; at 400 MB/s, about 47 minutes.

Runpod's own guidance of every hour to every few hours fits that. For a LoRA run the save is seconds, so save every 15 minutes: 12.45 s every 900 s is 1.4% of the run.

What does one stop cost on Runpod versus Nodus?

We priced one run four ways. It is an 8 x NVIDIA H100 SXM job that needs 10 hours of compute, checkpoints every 60 minutes and gets stopped once. We assume the worst case: the stop lands just before a save, so a full hour is redone, plus 10 minutes to restart and restore. That makes 11 hours 10 minutes, or 11.167 hours, of billed time when nothing saves at the stop.

Prices checked November 14, 2026, from the Runpod pricing page (Secure Cloud list rate; Community Cloud "from" rate in its FAQ) and the H100 SXM rate listed at $2.60/hr on the Nodus pricing page. Restart time and the timing of the stop are our assumptions. Runpod on-demand Pods are not reclaimed, so the last column does not apply there; a planned-maintenance email lets you save by hand.

For Runpod Secure Cloud, 8 x $4.39 = $35.12 per hour, and 11.167 x $35.12 = $392.17. On Nodus, 8 x $2.60 = $20.80 per hour, and 11.167 x $20.80 = $232.27. When the stop comes with a reclaim notice and the urgent checkpoint commits, only the 10 minutes of restore remain: 10.167 x $20.80 = $211.47.

At list prices, one badly timed stop costs about $41 on Runpod Secure Cloud and about $24 on Nodus; a notice save cuts the Nodus loss to $3.47.

Dollars are not the only cost. On Runpod, moving a run off a failed machine is your job: Runpod's docs have you deploy another Pod, so the run waits until someone does. On Nodus the replacement is prepared during the notice and started automatically.

How does Nodus keep an interrupted run alive?

Nodus treats every run as a series of attempts and moves the run, not the machine. When interruptible capacity gives a reclaim notice, Nodus asks your program for an urgent checkpoint and starts preparing a replacement at the same time (attempts and recovery). The replacement starts only once the old machine is provably gone, so two machines never write the same state. A checkpoint committed during the notice still counts.

Machines that fail without notice are covered too. A machine that stops sending heartbeats gets a replacement prepared after 20 seconds and, on shared capacity, is declared lost after 60. The new attempt restores your state directory and runs your command again, which loads the checkpoint as usual. Recovery stops with a typed reason, such as NoProgress after two attempts in a row with no progress, instead of looping and billing forever.

Your program writes to /nodus/state and loads from it on start, the same pattern as on Runpod (checkpoint guide). With interval: auto, Nodus sets the save cadence from how often the capacity is interrupted and how long your saves take, giving a run at least four checkpoints. From the docs:

pip install nodus-compute nodus login nodus run --gpu H100:8 --interruptible --checkpoint /nodus/state --max-cost 260 -d -- torchrun train.py 

The --max-cost 260 cap sits above the $232.27 worst case from the table, so one stop will not end the run early.

Can Nodus use the Runpod credits you already have?

Yes, in Beta, by placing work in a Runpod account you connect. In the console, open Cloud accounts, choose Runpod, paste an API key and set a credit limit (cloud accounts). Nodus then places single-node Jobs in your account first, within that limit. Runpod bills you for the machine and disk, and Nodus adds a Route fee of $0.02 per device-hour.

Each attempt's page says whether it ran in your account or on Nodus capacity. When the credit limit is used up, the default setting continues the work on Nodus capacity at Nodus rates. You can instead make it wait or ask you first.

When is Runpod the better choice?

Runpod's on-demand Pods are a good fit in several real cases:

• You want one machine nobody can take back. An on-demand Pod cannot be displaced by other users, which matters for a huge full fine-tune whose checkpoint takes many minutes to write.
• You want a long-lived box with /workspace, SSH and Jupyter that you treat like a workstation.
• You hold Runpod credits. Spend them directly, or route Nodus Jobs into that account.
• You need the H200 on Community Cloud, which the Runpod pricing page lists from $3.59/hr, a fraction of a cent below the $3.593/hr listed on the Nodus pricing page.
• You need a multi-node cluster today. Runpod sells Clusters for multi-node workloads; multi-node training on Nodus is in Beta.

Pick Nodus if you want interruptible capacity at a lower list rate with the notice handling, replacement and restore done for you. For a wider comparison, see Runpod alternatives in 2026 and how often spot GPUs are interrupted.

FAQ

Do old Runpod spot Pod scripts still work?

No. Runpod's v2 API has no equivalent for interruptible or podBidResume, and its REST API v1 is scheduled for retirement on November 15, 2026 (migration guide). Rewrite them to create on-demand Pods, or move the interruptible part of the work elsewhere.

Does a stopped Runpod Pod keep billing?

Compute stops, storage does not. Runpod charges $0.20/GB/month for a stopped Pod's volume disk, double the running rate, and network volumes keep billing at their normal rate (Pod pricing). Delete stopped Pods whose checkpoints you have already copied out.

Can I write checkpoints to a Runpod global volume instead?

You can, but Runpod says writes can be slower than on a network volume and suggests a network volume for fast writes (storage options). Global volumes are in beta and suit read-heavy files such as model weights.

Does Nodus resume a reclaimed run mid-step?

For some one-GPU Jobs, yes, in Beta. With GPU memory snapshots enabled, Nodus snapshots the stopped process and can resume it on a matching machine from the step that was running (checkpoint guide). Otherwise it restarts your command from the latest checkpoint.

Originally published on the Nodus blog.