The GPUs are available in one cloud. The training data lives in another. That mismatch is now routine: GPU capacity is scarce and unevenly distributed, while datasets were collected wherever the product happened to grow up. So a team books capacity where it exists, and then discovers the real project is not the training run. It is moving, and keeping moved, the data the training run feeds on.

The naive answer is a full copy before every run. Copy the dataset across, train, copy the results back, repeat next month with a slightly larger dataset. It works once. It fails as a habit, because most of the bytes never changed, the copy window grows faster than the training window, and egress charges quietly become a line item nobody budgeted for. Data gravity is not a metaphor to admire. It is an engineering constraint to design around.

Move the delta, not the dataset

The first fix is to stop treating the dataset as one indivisible object. A training dataset is a collection of items, most of them stable, some of them new, a few of them corrected or withdrawn. If the storage layer can describe the dataset as a manifest of items with content hashes, the cross-cloud question changes from "copy everything" to "which items are missing or different on the other side".

Sync by manifest. The receiving side lists what it already holds, compares hashes, and pulls only the missing or changed items. The first sync is still a full move, and there is no way around paying it once. Every sync after that is proportional to how much the data actually changed, which for most real datasets is a small fraction per training cycle.

Content addressing does quiet extra work here. If two datasets share items, and most training sets are revisions of earlier ones, the shared items transfer once across all of them. A corrected label is a new item, not a new copy of its neighbours. Deduplication stops being a storage feature and becomes a network feature.

Keep one golden copy and say where it lives

Delta sync only works if everyone agrees which copy is authoritative. Pick one home for the dataset of record, in one cloud, with one owner. Other copies are caches with a manifest and a freshness state, not rival sources of truth.

That distinction matters most for deletions and corrections. A privacy deletion or a labelling fix has to propagate to every cache, and the training pipeline should be able to prove which manifest version a run used. Record the manifest version in the run metadata, next to the code version and the hyperparameters. A model whose training data cannot be named exactly cannot be reproduced, and cannot be fully trusted when a question arrives months later about whether a particular record was in it.

Withdrawn items deserve special handling. Mark them in the manifest, remove them from caches before the next run, and verify the removal rather than assuming the sync handled it. Additions are easy to notice when they are missing. Deletions are easy to miss precisely because nothing breaks.

Put a cache near the GPUs, and treat it as a cache

GPUs are the expensive, scarce resource in this design, so the network should serve them, not the other way round. Stage the working set on storage in the same cloud, ideally in the same region, as the training capacity, and feed the run from there. A training job that reads its data across a cross-cloud link will spend GPU hours waiting on a network path that was never built for sustained bulk reads, and the bottleneck will look like a training problem in every dashboard except the network one.

Treat the staging copy honestly as a cache. It can be rebuilt from the golden copy. It does not need the same backup regime, and it should not quietly become a second home for data that exists nowhere else. Checkpoints and trained model artefacts are the opposite case: those are outputs that exist nowhere else until they are copied back, so they get the careful treatment, and the dataset cache gets the disposable treatment. Teams regularly invert the two, protecting the copyable data and leaving the irreplaceable outputs on temporary storage.

Size the cache by the working set, not by the archive. If a run trains on the last ninety days plus a fixed reference set, stage that, not ten years of history. A smaller, well defined working set syncs faster, verifies faster, and gives the deletion process a smaller surface to check.

Verify integrity before the run, not during it

A partial sync discovered at hour three of a training run wastes the scarcest resource in the system. Verify before the run starts: item counts and hash spot checks against the manifest, a read test on a sample from each part of the dataset, and a check that the staged copy reports the manifest version the run expects.

Make the verification cheap enough to run every time. Full hash verification of a large dataset on every run can cost as much time as the sync saved, so verify in layers: counts and sizes always, hashes on a rotating sample, and a full verification on a schedule and after any storage change. The point is not perfection. The point is that a mismatch surfaces in a pre-flight check that costs minutes, not in a failed run that costs a booked GPU window.

Count the movement, because someone is paying for it

Cross-cloud data movement has three costs that belong in the design, not in a surprise invoice. Egress charges apply to bytes leaving the source cloud, and a full copy per run multiplies them by the number of runs. Time is a cost: a sync window that grows with the dataset eventually collides with the training schedule. Attention is a cost: a manual copy step is a step someone forgets, rushes, or runs against the wrong version.

Delta sync, a regional cache, and manifest verification attack all three. The egress bill becomes proportional to change. The sync window becomes predictable. The manual step becomes a checked, repeatable job. None of this requires exotic tooling. It requires treating dataset movement as part of the training system, designed once, rather than as a favour the network does before each run.

A short checklist for cross-cloud training data

  1. One golden copy, one owner, one named cloud of record.
  2. A manifest of items with content hashes, versioned like code.
  3. Sync by delta against the manifest, never a blind full copy.
  4. A disposable cache in the GPU region, sized to the working set.
  5. Deletions and corrections propagated and verified, not assumed.
  6. Manifest version recorded in every run, next to code and parameters.
  7. Pre-flight verification in layers, before the GPU window opens.
  8. Checkpoints and outputs copied back with more care than the cache.

The pattern generalises beyond training. Any workload that follows scarce capacity into another cloud, inference fleets, rendering, large scale simulation, inherits the same gravity problem and the same answer: move what changed, cache what is read, protect what cannot be recreated. Teams that build that discipline into their private connectivity and data paths, rather than improvising a copy before each run, keep the expensive hardware fed and the data bill boring. That approach to AI workload connectivity across clouds is described at cloudverg.com, where VergMind and VergMesh are used to keep model traffic and the data behind it on defined, private paths.

The GPUs will keep living wherever capacity is available. The data does not have to chase them in full, every single time.