How many GPUs do you need to train an AI model?

Most fine-tunes need one GPU, full training of a 70B model needs 8 to 24, and pretraining needs as many as your deadline demands. On Nodus you ask for exactly that shape, from one RTX 4090 listed at $0.29/hr to several 8-GPU machines in one multi-node run (in Beta). The number to ask for is the larger of two floors: enough GPU memory to hold the training state, and enough GPU-hours per day to finish before your deadline.

We built this rule from published memory and compute formulas, and every table below is our own arithmetic, shown so you can redo it for your model. Our posts on VRAM for 70B models and how long training takes go deeper on each floor.

How do you work out how many GPUs a training run needs?

Compute two numbers and take the larger one. The memory floor is the fewest GPUs that can hold the model, gradients and optimizer state. The time floor is the fewest GPUs that can do the run's math before your deadline.

memory floor = training state (GB) / usable memory per GPU (GB) time floor = GPU-hours for the run / hours until the deadline GPUs to ask = max(memory floor, time floor), rounded up to 1, 2, 4 or 8, then to whole 8-GPU machines above 8 

The rounding follows how machines come. On Nodus a single machine gives you 1, 2, 4 or 8 GPUs, per the GPU guide, and anything bigger is several machines with the same GPU count each.

Fine-tuning is almost always decided by memory, because it sees few tokens. Pretraining is almost always decided by time, because it sees billions. Knowing which floor binds tells you what to change: a bigger GPU fixes the first, a later deadline fixes the second.

How many GPUs does the model need just to fit?

Count bytes per parameter, then divide by what each GPU can spare. Full training with Adam in mixed precision keeps about 16 bytes per parameter, per the ZeRO paper: 16-bit weights and gradients plus an FP32 master copy and two Adam moments. LoRA freezes a 16-bit base at 2 bytes per parameter and trains small adapters. QLoRA stores the base in 4 bits.

Activations, buffers and the CUDA context also need room, so we plan on 80% of each GPU's memory. That is 64 GB of an 80 GB NVIDIA H100 and 144 GB of a 180 GB NVIDIA B200. The 70B figures come from our 70B VRAM guide, which works them out for Llama 3.3 70B.

Our arithmetic: training state divided by usable memory, rounded up. Activations grow with batch size and sequence length, so treat these as minimums.

A GPU with more memory shrinks the count: full 70B training goes from three H100 machines to one B200 machine.

Two rows sit right on the line. The 8B full run needs exactly 2 H100s at 64 GB each, with nothing spare, so we would ask for 4. The 70B LoRA run fills one B200 exactly; our VRAM guide finds it still fits with one 4,096-token sequence, but longer sequences need a second GPU.

Spreading a model over several GPUs needs sharding, such as PyTorch FSDP or DeepSpeed ZeRO-3, so each GPU holds only its share. Our multi-GPU PyTorch guide shows the code for both DDP and FSDP.

How many GPUs do you need to finish by a deadline?

Divide the run's GPU-hours by the hours you have. Training a transformer costs about 6 floating-point operations per parameter per token, from Kaplan et al.. An H100 SXM peaks near 989 teraFLOPS of dense BF16, half the sparse figure on NVIDIA's H100 page. At a 30% utilization planning figure, one H100-hour does about 1.07 x 10^18 useful operations.

Take a 7B model pretrained on 140 billion tokens: 6 x 7 billion x 140 billion is 5.88 x 10^21 operations, or about 5,505 H100-hours. Here is the GPU count that deadline math gives.

Our arithmetic at 30% utilization and perfect scaling. Divide the time floor by your measured scaling efficiency: at 90%, the 14-day row needs 18.2 GPUs, which still rounds to 24.

The 3-day row needs ten machines. That is more than the 8-machine Beta ceiling for a Nodus multi-node run, so on Nodus today that deadline means a bigger GPU or a later date.

Halving the deadline roughly doubles the GPU count, while the GPU-hours stay near 5,505.

The bill barely moves with the count. At $2.60 per H100-hour, listed on the Nodus pricing page, 5,505 GPU-hours is about $14,313 whether you use 8 GPUs or 40. More GPUs buy calendar time, and the real cost creeps up only by the scaling loss.

Fine-tuning shows the opposite. A full fine-tune of an 8B model on 50 million tokens is about 2.2 H100-hours by the same formula, so one GPU finishes overnight. Its time floor is 1, so the memory floor decides: 2 GPUs, or 4 with headroom.

Which GPU should you count in?

Count in the GPU whose memory makes the memory floor fit on one machine, then compare hourly cost. One machine avoids the network between machines, which is slower than NVLink inside one and is another place for a run to fail.

Full training of the 70B model shows why. On H100s it needs 24 GPUs across three machines, or $62.40 an hour at $2.60 each. On B200s it needs 8 GPUs on one machine, or $44.00 an hour at the $5.50 listed rate.

The 24 H100s do have more peak math, about 23.7 dense BF16 petaFLOPS against 18 for an 8-GPU B200 board (NVIDIA figures, compared in our B300 vs B200 post). Per dollar the B200 still edges ahead, 0.41 against 0.38 petaFLOPS per dollar-hour, and it skips the network between three machines.

Where an H100 count spills past one machine, the bigger GPU is cheaper per hour as well as simpler.

The cheaper GPU wins when the job fits it. An 8B LoRA run fits one 24 GB card, and an RTX 4090 is listed at $0.29/hr against $2.60 for an H100. Prices checked November 5, 2026, on the Nodus pricing page.

There is also a practical limit on Nodus today. The multi-node guide puts the Beta ceiling at 8 machines, and each pair of providers has its own qualified member cap. It also notes that a two-machine DDP result does not yet qualify FSDP across machines, so a sharded run over several machines needs a check of the exercised combinations first. That makes "fit it on one machine" the safer rule for now.

How do you check the number before paying for it?

Measure throughput at two or three GPU counts, then let real numbers replace the 30% guess. A Nodus Sweep runs the same Job across a matrix of GPU requests and records cost, wall time and throughput for each cell, per the Sweeps guide. The matrix takes requests such as H100:8, so one Sweep can cover 1, 2, 4 and 8 GPUs on one machine.

 matrix: gpu: ["H100:1", "H100:2", "H100:4", "H100:8"] 

Each cell reports costUSD, wallSeconds and unitsPerSecond if your script logs units, and status.best names the cheapest and fastest cells. If throughput at 8 GPUs is less than about 7 times the 1-GPU figure, your scaling efficiency is under 90%, and you should plan with that number.

For a run that needs several machines, use the dry run from the multi-node guide. It shows the shape Nodus picks and the hold for each machine before you commit:

$ nodus run --dry-run --gpu H100 --total-gpus 16 --launcher torchrun -- torchrun train.py 

FAQ

Can you train an AI model on a single GPU?

Yes, if it is a fine-tune or a small model. Every LoRA and QLoRA job in the memory table fits one B200, and a 70B QLoRA run fits one 80 GB GPU. Pretraining a 1B model on 20 billion tokens is about 112 H100-hours, nearly 5 days on one GPU.

How many GPUs did Meta use to train Llama 3?

Up to 16,384 H100s, according to the Llama 3 paper. The same paper reports 43% utilization on 8,192 GPUs and 41% on 16,384, a measured example of the scaling loss you pay for more GPUs.

Do I need the same GPU type for every machine in a run?

Not on Nodus. If you list several types, the multi-node guide says members may get different ones. Every member still gets the same GPU count, and the shape is fixed for the whole run, so your world size never changes on a restart.

Should I ask for more GPUs than the floor to be safe?

Ask for headroom in memory, not in count. Running out of memory crashes a run, so round the memory floor up generously. Extra GPUs past the time floor only finish sooner at the same or higher cost.

Originally published on the Nodus blog.