The most expensive way to discover a slow network is with a fleet of accelerators already allocated. The job starts, the GPUs sit at low utilisation, the training framework reports that it is waiting on data or on peers, and the meter runs anyway. By the time someone proves the bottleneck is a storage path in another cloud, the run has burned hours that a ten minute check would have saved.
Training jobs are usually scheduled as if compute were the scarce resource and everything else were already true. Across clouds, everything else is exactly what is not guaranteed. The dataset lives in one place, the accelerators were available in another, the checkpoint bucket is in a third, and each of those paths behaves differently under load than it does in a diagram.
A preflight check treats the network and storage paths as part of the job's requirements, tested before the scheduler releases the expensive part.
What actually starves a training job
Three paths decide whether accelerators stay fed, and each fails quietly:
- The data path. Readers pull shards from object storage or a parallel file system, often in a different cloud or region. A path that delivers fine for one reader can collapse when sixty four readers start at once, because the bottleneck is per connection throughput, request rate limits, or a shared link, not headline bandwidth.
- The peer path. Distributed training exchanges gradients or activations between workers. That traffic is synchronised, so the slowest participant sets the pace for everyone. One worker placed across a congested or long detour path makes the whole job wait at every step.
- The checkpoint path. Writes are bursty and large. A checkpoint that takes forty minutes instead of four does not just waste time, it extends the window in which a failure loses work, and it often competes with the data path on the same link.
None of these show up in a readiness probe that only checks that endpoints answer. Reachable is not the same as fast enough, and fast enough for one node is not fast enough for the full job shape.
Run the preflight at job shape, not at laptop shape
A useful preflight mirrors the real run in miniature, with the same counts and patterns where it matters:
- Test storage reads with the same number of concurrent readers the job will use, reading the actual shard format at the actual block size. Record sustained throughput per reader and in aggregate, not a single file download time.
- Test the peer pattern, not just point to point bandwidth. A simple all reduce or broadcast test between the actual worker placements reveals whether one placement is an outlier before the framework spends an epoch proving it.
- Test a checkpoint write of realistic size to the real destination, and time it end to end, including the close and consistency steps the framework waits on.
- Resolve and connect to every endpoint the job will touch, from the actual worker network, including private endpoints and their DNS answers. Cross-cloud private names are a common place for a job to discover, at startup, that it can only reach half its services.
- Check the quiet quotas: egress limits, request rate ceilings, and concurrent connection limits on the storage and network services in between. Jobs rarely fail loudly on quotas. They slow down and look like a framework problem.
Each check gets a threshold written down in advance, tied to the job's own arithmetic. If the model needs a given data rate per accelerator to stay fed, the preflight pass mark is that rate with headroom, not a vague sense that the numbers looked good.
Fail before the allocation, not after
The point of thresholds is a decision. When a check fails, the scheduler should hold the job and say which path failed and by how much, while the accelerators are still available for work that can use them. Three responses cover most failures:
- Move the job. If the peer test shows one placement lagging, placing all workers in the cloud where the capacity actually performs may beat the original plan, even if the per hour price is higher. Idle accelerators are the most expensive kind.
- Move the data, once. If the data path cannot sustain the read rate, staging a copy near the compute before the run, and verifying it against a manifest, converts a continuous network dependency into a one time transfer.
- Shrink the job shape. Fewer workers in the place that performs can finish sooner than more workers waiting on each other across a slow boundary. Synchronised work rewards locality more than raw count.
What does not help is starting anyway and hoping the framework adapts. Frameworks report symptoms faithfully and fix none of them.
Make preflight a habit, not an incident step
Paths drift. A route changes, a storage tier fills, a new tenant shares the link, and last month's healthy path is this month's bottleneck. So the preflight belongs in the normal launch path, run every time, with results recorded next to the run they gated. When a job later underperforms, that record answers the first question immediately: did the paths pass that morning, and at what numbers?
Keep the checks fast enough that nobody is tempted to skip them. Ten minutes of targeted tests is cheap insurance against hours of allocated idle time, and a preflight that is routinely skipped is worse than none, because it creates confidence without evidence.
The design point
Accelerators are rented by the hour and networks are paid for in waiting. Before a cross-cloud training job starts, prove that data can arrive at the rate the model consumes it, that workers can talk at the pace synchronisation demands, and that checkpoints can leave before they become the schedule.
That kind of placement testing and path design across AWS, Azure, Google Cloud, and Oracle estates is part of the AI infrastructure connectivity work described at CloudVerg, where VergMind covers AI workload connectivity and VergMesh covers the underlying mesh, alongside zero-trust security and SOC 2 readiness support.
GPUs waiting on a network are not a compute problem. They are a scheduling decision that a short preflight can make honestly.

