I rented an NVIDIA Blackwell pod on RunPod. Spun up the environment. PyTorch imported cleanly. torch.cuda.is_available() returned True. Everything looked fine.
Then I ran uvx computefence doctor.
✗ NVIDIA RTX PRO 4000 Blackwell has compute capability sm_120, which is not supported
by this PyTorch build (supports up to sm_90)
Fix: Install a PyTorch build that supports sm_120: see https://pytorch.org/get-started/locally/
Exit code 2. BLOCKER. Job stopped before it started.
What was about to happen
The pod had an NVIDIA RTX PRO 4000 Blackwell compute capability sm_120. My installed PyTorch was 2.4.1+cu124, which only includes compiled kernels up to Hopper (sm_90).
PyTorch doesn’t flag this at import time:
import torch succeeds cleanly.
torch.cuda.is_available() returns True because the GPU is visible and the CUDA runtime loads.
Your script downloads weights, tokenizes datasets, and warms up burning billable compute the whole time.
The crash only triggers when PyTorch executes its first CUDA kernel on a forward pass:
RuntimeError: CUDA error: no kernel image is available for execution on the device
Pod still billing. Model not trained. No checkpoint saved.
Why torch.cuda.is_available() isn’t the right check here
There are four separate things that need to work:
GPU visibility: Can the OS and NVIDIA tooling see the GPU?
CUDA availability: Can PyTorch initialise CUDA and access the device?
Architecture compatibility: Does your PyTorch build contain compiled kernels for the GPU’s compute capability?
Kernel execution: Can your training workload actually run on this hardware?
torch.cuda.is_available() only confirms the first two. You can verify the mismatch manually:
python
import torch
cap = torch.cuda.get_device_capability()
print(f"GPU architecture: sm_{cap[0]}{cap[1]}")
print(f"Supported by PyTorch: {torch.cuda.get_arch_list()}")
If sm_120 isn’t in that list, the job is a ticking clock.
Why this is becoming more common
RunPod and Vast.ai are adding Blackwell nodes because they are fast and cost-competitive. Most existing Docker templates and install commands still pull cu118, cu121, or cu124 builds. None of those wheels contain sm_120 binaries Blackwell requires cu128 builds (PyTorch 2.7+ or nightly).
Most people only discover the mismatch after the job crashes.
How to fix it
Force-reinstall PyTorch with a CUDA 12.8 wheel:
bash
pip install --force-reinstall torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
Use --force-reinstall a plain pip install torch often does nothing if the version number already appears satisfied.
Catch it automatically before the job starts
Remembering to check get_arch_list() every time a provider assigns a newer GPU is easy to forget especially when is_available() already returned True.
bash
uvx computefence doctor && python train.py
If computefence doctor exits with a non-zero code, train.py never runs. Two minutes fixing the wheel instead of 20 minutes paying to watch a model load and crash.
The pattern
This is the same failure others have hit on Blackwell pods GPU assigned, older CUDA wheel installed, job dies at the first kernel with a cryptic error. PyTorch gives no warning until after you’ve paid for the warmup.
The check exists now. Run it before the job.
pip install computefence · uvx computefence doctor