NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale
Authors: Songlin Jiang, Zhiyu Li, Terry Kong, Yu Yao, Youngeun Kwon, Bernard Nguyen, Ashwath Aithal, Mario Di Francesco (NVIDIA)
Abstract
Abstract. Agentic reinforcement learning (RL) disaggregates training from rollout, so each policy update must reach the rollout clusters before the next batch. Transferring a full 1T checkpoint for such weight synchronization (refit) takes 87.5 min between two AWS regions. Measurements of BF16 training show that about 1% of weights change their stored values per step. Recent systems exploit this sparsity but fall short on placement, exactness, or efficiency: they reimplement placement rules, assemble full tensors, rebuild values arithmetically, or use a cross-cluster collective, and none fully recovers from mid-refit failures. We present NeMo-DCR (Delta-Compressed Refit), which sends only changes yet is bit-exact: receivers obtain the same parameter and buffer bits as a dense refit. For placement, fixed affine mappings project changes from training shards into the checkpoint’s canonical coordinates, residual conversion covers the other changes, and the serving runtime’s native loader places all changes in receiver storage. For exactness, compressible XOR masks carry affine changes whose projection and loader preserve stored bits, and overwrites carry the others. Receivers apply both in place, retries overwrite partial writes, and a joint commit binds the policy to the baseline for the next delta. For efficiency, object storage or a relay tree streams payloads during delta construction, without a cross-cluster collective. Even at 3% and 5% change rates, NeMo-DCR refits of 30B–1T models are 12–40× faster than a transport-only full-checkpoint reference. A 1T relay-tree refit at 3% takes 150 s instead of 87.5 min, making refits practical for cross-cluster agentic RL at trillion-parameter scale. Documentation: NeMo RL weight refit guide Code: NeMo RL pull request #2444
1 Introduction
Agentic reinforcement learning (RL) spends over 70% of its wall-clock time in rollout, so rollout is disaggregated from training and runs on serving clusters with suitable hardware and parallel layouts [6, 7]. Before the next rollout batch, a refit must deliver the updated policy to the receivers, the rollout ranks that generate responses with a serving runtime [15]. Disaggregation also lets rollout run on idle serving GPUs [6] or in other datacenters [41, 40], where every refit must cross a wide-area network.
A cross-cluster transfer of a 120B model’s 247.2 GB checkpoint through object storage takes 750 s, and at the 1T scale of models like Kimi K2 [14], such a transfer takes 87.5 min (Figure 1). Yet in the six models of Figure 2, only 0.6–1.2% of training-side source elements change their bfloat16 (BF16) values per step. We call this fraction the element change rate. With only 8 significant bits, BF16 rounds most optimizer updates away. Other reports give element change rates of 1.5–2.0% in Qwen3 RL [26] and average deltas of 1.98% of a 1,024 GiB checkpoint [5]. A full-checkpoint transfer thus spends 98–99% of its bytes on unchanged elements.
Figure 1: NeMo-DCR relay-tree refit at a 3% element change rate versus the transport-only full-checkpoint reference: 22.6 s instead of 750 s at 120B and 150 s instead of 87.5 min at 1T (§§ 8.5 and 8.6).
Figure 2: Element change rate per optimizer step, averaged over each model’s first five steps with the setup of Appendix E: at most 1.2% in every model.
(Footnote: footnotetext: Work conducted during an internship at NVIDIA.)
A refit that sends only the changed 1% faces three challenges: placement, exactness, and efficiency.
First, placement: sparse source changes do not tell receivers where or what to write. A training-shard location does not map directly to a receiver’s storage location: refitting may gather, split, or permute tensors between training and rollout layouts [26]. A change to a shared scale factor can also alter converted values even when other source values stay unchanged [23]. Some systems reimplement the serving runtime’s placement rules [6, 26], and others extract a delta after full-tensor assembly and conversion, which handles both problems but pays for both steps [11].
Second, exactness: a smaller transfer must still deliver the exact updated bits, because rollout–training mismatch is known to destabilize RL [28]. Arithmetic reconstruction can introduce floating-point rounding errors [26], while absolute overwrites avoid rounding [19] but resend unchanged bits within changed values. Failures also threaten exactness: applying updates in place avoids a separate receiver-side baseline copy, but an interrupted refit can leave a mixture of old and updated values that matches neither policy version. Because a delta applies only to the version it was computed from [19], recovery must complete the interrupted refit before rollout resumes and keep receivers aligned with the source baseline that the next refit compares against.
Third, efficiency: delivery must overlap the other refit stages without a cross-cluster collective. The next rollout batch waits for the complete update, so without overlap, delta construction and application can offset the transfer savings. A cross-cluster collective couples receiver membership to the training cluster and introduces timeouts upon failure [22]. Some deployments share weights only through object storage [3], and others have direct links between clusters [40], so delivery must support both.
Existing systems meet these challenges only in part: they either send every weight or lack exactness, native placement, decoupled delivery, or recovery, so none meets all five requirements in Table 1. We present NeMo-DCR, a system for bit-exact delta refit that meets all five: it sends only changes, yet every receiver obtains the same parameter and buffer bits as a dense refit that sends every weight. The key idea is to describe changes in canonical coordinates, the tensor names and indices of the Hugging Face checkpoint format that both training and serving already support. Fixed affine index mappings project changes directly into these coordinates and cover over 96% of the weight bytes in the mixture-of-experts (MoE) checkpoints. The serving runtime’s native loader then maps these coordinates to receiver storage, much as a page table maps virtual to physical addresses.
| Approach | Systems | Bit-exact | Delta | Native | Decoupled | Recovery |
|---|---|---|---|---|---|---|
| G1 | G2 | G3 | G4 | G5 | ||
| A Dense collective refit | NCCL-Reshard [22] | ✓ | – | (✓) | – | (✓) |
| checkpoint-engine [20] | ✓ | – | ✓ | – | (✓) | |
| B Full-checkpoint transfer | TensorHub [40] | ✓ | – | (✓) | ✓ | (✓) |
| C Arithmetic reconstruction | ROSE [6] | – | ✓ | – | ✓ | – |
| AReaL-DTE [26] | – | ✓ | (✓) | ✓ | (✓) | |
| AuroraRL [30] | – | ✓ | – | ✓ | (✓) | |
| D Absolute overwrites | PULSE [19] | (✓) | ✓ | – | ✓ | (✓) |
| SparseRL-Sync [11] | (✓) | ✓ | ✓ | – | – | |
| verl [36] | (✓) | ✓ | ✓ | – | – | |
| Bit-exact delta refit | NeMo-DCR | ✓ | ✓ | ✓ | ✓ | ✓ |
| ✓ means met, (✓) partly met, and – not met or not described in the cited source. Details: Appendix F. |
Table 1: Refit approaches scored against the five requirements of § 2.3. Only NeMo-DCR meets all five.
Building on canonical coordinates, this paper makes four contributions that address these three challenges:
-
- Direct projection and residual conversion. One training rank per shard acts as its owner, which detects changes in stored values and projects affine changes into canonical coordinates. Residual conversion compares converted tensors with a distributed residual baseline to cover the other changes, including those from shared scale factors. Avoiding full-tensor assembly and conversion for affine changes makes Qwen3 delta construction 1.08–1.16× faster at 3% and 5% element change rates (§§ 4 and 8.3).
-
- Mixed XOR/overwrite encoding with compression. XOR encodes non-overlapping affine changes whose projection and loader preserve stored bits, while overwrites encode all others. Both are exact, and mixed encoding cuts Qwen3 payload bytes by 38–40% versus overwrites (§§ 5, 8.2, and 8.3).
-
- Recoverable in-place refits. Receivers leave placement to the native loader and apply updates in place, keeping no separate receiver-side baseline copy. Retries overwrite the complete changed set, repairing partial writes. After all current receivers apply the updates, a joint commit binds the policy to its source baseline, which owners then update in place. Proposition 1 guarantees bit-exactness. With receivers killed mid-refit, both NeMo-DCR transports follow the mean reward and KL trajectories of dense NCCL refits (§§ 6, 8.1, and 8.4).
-
- Pipelined delivery without a cross-cluster collective. Payloads travel over object storage or a relay tree as the delta is built and, in synchronous RL, applied. In asynchronous RL, rollout continues serving requests until the delta has been transferred and staged, then pauses while receivers apply it. At 3% and 5%, relay-tree transport overlaps delta construction, and the transport lower bound accounts for 77–94% of the refit latency (§§ 7 and 8.7).
Together, these contributions make NeMo-DCR refits of 30B–1T models 12–40× faster than the transport-only full-checkpoint reference (§§ 8.5 and 8.6), even in stress cases whose 3% and 5% element change rates exceed every rate in Figure 2. The same code handles Qwen3 and hybrid Mamba-Transformer Nemotron models without model-specific logic. At 1T and 3%, the relay tree takes 150 s instead of 87.5 min (Figure 1), making refits practical for cross-cluster agentic RL at trillion-parameter scale.
2 Motivation and Requirements
2.1 Refit across layouts and versions
Training systems and serving runtimes store the same policy in different layouts. Training systems shard parameters, gradients, and optimizer state to fit into GPU memory while maximizing update throughput [21, 29]. Serving runtimes organize weights for inference and maintain request state such as a paged key-value (KV) cache [15]. RL systems that coordinate the training and rollout sides must bridge the two layouts [32, 10, 13]. The two sides choose their parallel degrees independently, so one training shard can overlap several rank-local slices on receivers, and one slice can combine parts of several shards (Figure 3).
Figure 3: Top: one tensor in the training, canonical, and rollout layouts. Bottom: BF16 values near 0.5, labeled by their stored bits in hexadecimal. Values in the shaded interval round to 0x3F01, and only an update crossing its boundary changes the stored bits.
Both layouts map to canonical tensors: the model’s tensors in the Hugging Face checkpoint’s naming and layout, independent of either cluster’s sharding. Let Wv denote the canonical tensors at version v, including versioned buffers. A receiver r does not store Wv directly. Its native loader Lr is the serving runtime’s own weight loader and may split, combine, skip, transpose, or cast tensors before writing rank-local storage Prv=Lr(Wv). For example, vLLM fuses the query, key, and value projections and splits routed experts across ranks. Each sequence of such operations is a loader path, and we consider loader paths whose output elements each come from one canonical element.
A refit must bring each receiver to Prv+1=Lr(Wv+1), the result of a dense refit. Dense collective refit and full-checkpoint transfer produce this result by sending every weight, and Table 1 lists them as approaches A and B. We call the update to version v+1 a transition, which may take several attempts. Each attempt updates the current receivers, called its required receivers.
2.2 Few BF16 values change per step
How many values a refit must update depends on training: an update changes a stored BF16 value only when it crosses a rounding boundary (Figure 3), so larger learning rates and early training steps change more stored values. Yet even in the first five steps of Group Relative Policy Optimization (GRPO) [31], the six models in Figure 2 show only 0.6–1.2% average element change rates, counting each unique source element once. Appendix E details the setup, which follows an agentic software-engineering (SWE) RL recipe with a constant 10−6 learning rate.
2.3 Requirements and related work
To exploit this sparsity, a refit must meet five requirements that make placement (G3), exactness (G1 and G5), and efficiency (G2 and G4) concrete (Table 1). G1 Bit-exact: starting from receiver storage Prv, a delta refit must produce the same parameter and buffer bits as a dense refit. G2 Delta: transfer volume must scale with the emitted changed values, their locations, and payload metadata. G3 Native: the native loader must determine placement, with no model-specific placement rules in the refit system. G4 Decoupled: no collective may span the training and rollout clusters. G5 Recovery: receivers must suppress duplicate payloads, partial writes must be recoverable, and one authoritative commit record must bind each new policy version to its source baseline after every required receiver holds its new storage Prv+1. No existing system in Table 1 fully meets G5, and each fully meets at most two of the five requirements.
Other systems that ship policy weights to serving runtimes are limited in scalability or efficiency: they send full checkpoints, rebuild full weights from deltas, or support only unsharded runtimes. INTELLECT-2’s SHARDCAST relays checksummed full checkpoints in pipelined shards [27]. In Composer 2 and Fireworks, trainers upload per-step deltas to shared storage, from which inference clusters rebuild full weights [3, 5]. TRL’s delta sync likewise rebuilds full tensors from a CPU snapshot [12], and the slime framework applies XOR or overwrite deltas to a full local checkpoint and reloads it at every refit [33]. vLLM patches runtime parameters in place, but doesn’t support any parallelism [37].
Related techniques target training traffic, fine-tuning, file transfers, or storage rather than policy refits: gradient compression [17, 38], low-rank adaptation [9], delta file transfer and lossless weight compression [34, 8], and compression of fine-tuning deltas and checkpoints [18, 16, 39, 4].
3 Design Overview
Canonical coordinates split a refit into delta construction and receiver-side placement. Figure 4 numbers the seven steps: owners construct deltas, a transport delivers them, receivers apply them, and the coordinator, which drives each attempt, publishes the policy with a joint commit. The next four sections (§§ 4–7) explain these steps in detail and develop the four contributions in turn: projection and conversion, encoding, application and recovery, and delivery.
Figure 4: NeMo-DCR architecture and the numbered steps of a refit. The coordinator calls prepare to start an attempt and open_apply to let receivers apply payloads and then acknowledge (ACK).
Projection and conversion (❶–❷, §4). Shard owners compare their weights with the source baseline and directly project changes covered by affine mappings, which are fixed and convert each source element independently. The rest are residual: their canonical tensors must be constructed first (contribution 1).
Encoding (❸, §5). The codec encodes a change as an XOR entry when projection and the loader path preserve stored bits and no other write overlaps it, and as an overwrite entry otherwise (contribution 2). The first three steps together meet G1 and G2.
Application and recovery (❺–❼, §6). Each receiver validates and deduplicates payloads, then applies their XOR or overwrite entries in place through the native loader. Failed attempts are retried with overwrites. After every required receiver acknowledges, the coordinator’s joint commit binds the policy version to its source baseline (contribution 3). These steps together meet G1, G3, and G5.
Delivery (❹, §7). Object storage or the relay tree delivers the entries in payloads without a cross-cluster collective (G4). Streaming overlaps delivery with delta construction and, in synchronous RL, also with application (contribution 4).
Example. Suppose that a BF16 element changes its stored bits from 0x3F01 to 0x3F02 in a matrix sharded by rows in training and by columns in rollout. Its owner detects the change (❶), projects it to its canonical destination d (❷), and emits the XOR mask 0x0003 in a compressed payload (❸). The payload reaches every rollout node (❹), and each node’s receivers validate it (❺). Each receiver whose column slice holds d intercepts the loader’s final storage copy in order to XOR 0x0003 into 0x3F01 (❻). The joint commit publishes the updated policy, and the owner’s baseline becomes 0x3F02 (❼).
Implementation. NeMo-DCR is about 7K lines of Python. It builds on Megatron Bridge conversion [23] and the vLLM native loader [15].
4 Direct Projection and
Residual Conversion
In ❶–❷ of Figure 4, NeMo-DCR compares each training shard with its source baseline in place, on the shard owners: direct projection avoids full-tensor assembly and conversion for affine changes, and residual conversion covers the other changes.
The changes that an owner can directly project depend on the conversion tasks through which the Megatron Bridge library converts training shards to canonical tensors [23]. Affine tasks convert each source element independently and deterministically with the fixed affine index mappings of supported mapping classes, such as those for replicated and row- or column-split shards. Tasks for fused query/key/value (Q/K/V) source tensors and other tasks outside the supported mapping classes are residual and require constructing canonical tensors. Affine mappings cover 96.3% of the weight bytes in the Qwen3-30B-A3B checkpoint and 97.0% in the Nemotron-3-Ultra-550B-A55B checkpoint, both MoE models, as Appendix D details.
4.1 Ownership and change detection
In ❶, only one owner compares each unique source element against the source baseline, because data-parallel training keeps its replicas bitwise identical. A stable name hash balances ownership across data-parallel ranks (Figure A.2) and keeps it fixed across the attempts of a transition.
Owners detect changes against the committed source baseline Bv. It contains a task tracker for each conversion task, holding the version-v values of the task’s source shard, and a distributed residual baseline of canonical tensors for residual tasks. Because each unique source element has one owner, the trackers together hold one copy of the source weights. For task T, let STv+1 be the candidate shard, ATv its tracker, and JT its owned indices. Let I(X)j denote the stored bit pattern of X at j. The changed indices are
C_{T}=\{j\in J_{T}\mid I(S_{T}^{v+1})_{j}\neq I(A_{T}^{v})_{j}\}. Comparing stored bits detects all changes, even those that floating-point equality would miss (Figure A.2).
Baseline memory. The baseline uses host memory, not GPU memory. Beyond one copy of the source weights in the trackers, it holds the canonical tensors of residual tasks, which are 3.69% of the Qwen3-30B-A3B checkpoint and 3.01% of the Nemotron-3-Ultra-550B-A55B checkpoint. It thus totals about 1.04× and 1.03× the respective checkpoint. Balanced ownership spreads it across the training ranks: for the 1,121 GB Nemotron-3-Ultra-550B-A55B checkpoint, each of the 32 training ranks of § 8.1 holds about 36 GB.
4.2 Direct projection
In ❷, fixed shard geometry and index mappings let an owner compute the canonical destination of each change locally. For a changed index j of affine task T, the index mapping πT identifies the canonical destination, and the source projection QT produces its updated value:
d=\pi_{T}(j),\qquad o_{T}^{v+1}(j)=I(Q_{T}(S_{T}^{v+1}))_{d}. Here, d=(n,k) identifies a canonical tensor name and flat index, and oTv+1(j) is the updated stored value.
The owner emits each change once for all receivers without knowing their storage offsets. In the example of § 3, a changed local element (i,y) in a row shard that begins at row i0 maps to canonical destination (n,k), where k=(i0+i)w+y for w columns (Figure 5). Each receiver’s native loader then selects its own column slice.
Figure 5: Top: affine and residual changes. Bottom: XOR and overwrite entries of the example, each applied twice to 0x3F01.
4.3 Residual conversion
In ❷, residual conversion covers residual tasks, which fall outside direct projection because they lack a supported mapping, can spread a source change across several outputs, or combine inputs from several owners. Examples include stacking, padding, tied weights, adapters, custom postprocessing, and quantization scales. For these tasks, NeMo-DCR constructs the affected canonical tensors and compares their bits with the committed residual baseline (Figure 5).
Change detection covers every input of a conversion because a change in a shared scale factor λ can alter many converted values even when the other source values stay unchanged. For example, a fixed conversion rule may multiply each source value by λ and cast the product. Owners set a flag when a residual task’s shard changes (Figure A.4). Combined with known conversion dependencies, the flags select every canonical tensor affected directly or indirectly. The selected tensors form the required tensor set \mathcal{U}_{\mathrm{req}}. The delta omits outputs whose converted bits remain unchanged.
For each selected tensor u, the owners convert their parts, and a canonical-name hash selects the residual-baseline owner that assembles these parts (Figure A.4). The parts must jointly cover every element of u, and overlapping parts must agree bit-for-bit. Comparing the assembled result Wuv+1 with its committed residual baseline Ruv gives
D_{u}=\{(n_{u},k)\mid I(W_{u}^{v+1})_{k}\neq I(R_{u}^{v})_{k}\}, where nu is the canonical name. Residual tensors outside \mathcal{U}_{\mathrm{req}} keep their committed values.
Both paths feed one changed set: the emitted destinations combine direct projections πT(CT) and residual changes Du. Each emitted destination has a delta record with its updated value, and these records form the complete changed set \mathcal{E} that the codec encodes.
5 Mixed XOR/Overwrite Encoding
with Compression
In ❸, the XOR/overwrite encoding keeps reconstruction exact, and XOR also improves compression. Delta records in canonical coordinates share one codec and one payload format: the affine/residual path decides how a record is built, and the XOR/overwrite encoding decides how it is applied.
Exact encoding rules out arithmetic reconstruction, which can fail to recover the updated stored values: for an old value α and an updated value β, rounding can make α+(β−α) differ from β, and such errors can accumulate across versions. NeMo-DCR instead sends either XOR masks of stored values or absolute overwrites. Both reproduce the new bits exactly: a mask flips only the bits that differ, and an overwrite replaces the whole stored value.
For direct projection, the mask of changed index j is
x_{T}^{v+1}(j)=I(S_{T}^{v+1})_{j}\oplus I(A_{T}^{v})_{j}. However, the mask yields the target bits only on a representation-preserving path, where the source projection QT and native loader preserve dtype and stored bits. In the shared-scale example of § 4.3, multiplication and casting change stored bits, so a source mask does not carry over to the converted values (Figure D.1). Comparison with the residual baseline finds such changes after conversion, and absolute overwrites carry the converted bits.
To exploit unchanged bits within changed values, the default mixed mode uses XOR entries (d,x) wherever an affine path is representation-preserving and no other write touches the same bytes. These conditions are decided in advance for each mapping class and loader path (Figure D.1). All other records become overwrite entries (d,o). In the two MoE checkpoints of § 4, most conversions are affine and preserve stored bits, so XOR covers more than 96% of the weight bytes, as Appendix D details.
XOR entries also compress well: small updates often leave high-order bits unchanged, so XOR masks contain many leading zeros. In the example of § 3, the XOR mask 0x0003 has 14 leading zero bits (Figure 5). Run-length and gap coding shrink location bytes, and level-1 zstd [2] then compresses both values and locations in each payload.
Yet XOR makes duplicate delivery unsafe: applying a mask twice restores the old bits (Figure 5), so receivers suppress duplicate payloads by their identifiers. For the same reason, retries repair partial writes with overwrites.
6 Recoverable In-Place Refits
In ❺–❼, native-loader placement and overwrite recovery allow in-place refits from canonical coordinates without a receiver-side baseline copy. Reimplementing placement would duplicate the rules for fused Q/K/V, routed experts, slicing, padding, transposition, and tied weights. NeMo-DCR instead lets the native loader apply these rules, intercepts the loader’s final storage copies, and specifies loader conditions that guarantee dense-refit bits. Overwrite recovery repairs partial writes, and a joint commit keeps receiver storage and the source baseline in version alignment: each transition starts with every required receiver at the committed version.
6.1 Applying deltas through the native loader
Receivers apply payloads through the native loader one item at a time, where an item holds one owner’s entries for one tensor. After validating and deduplicating payloads (Figure B.2), each receiver scatters each item into a reusable host scratch buffer with the canonical tensor’s shape and dtype. Positions without an entry are inactive and hold a placeholder q (Figure 6, top). The loader resolves names and transforms the scratch, while a PyTorch dispatch hook updates only active positions at the loader’s final storage copies.
Figure 6: Top: applying one item in ❻ of Figure 4. Bottom: the chain of Proposition 1 and the property used at each arrow.
XOR path. Scratch holds the placeholder q=0 except for masks at changed positions, such as 0x0003 at d in the example. The intercepted copy XORs each mask into the matching resident element, and the copy’s input must share storage with scratch so the masks arrive unchanged. XOR with q=0 leaves inactive positions unchanged.
Overwrite path. Scratch holds overwrite values and a placeholder that must stay distinct from these values until the first intercepted copy into each destination view. At that copy, non-placeholder positions form the view’s boolean active mask of positions to update (Figure A.10). NaN is the default placeholder.
Receiver memory. Beyond resident weights, receivers keep only the scratch, staged payloads, and temporary buffers for active masks and optional post-apply checks of intercepted copies (Figures C.2 and A.12). A prewarm call reserves scratch for the largest canonical tensor and identifies the tensors that the native loader skips on each rank, such as routed experts held by other ranks.
Loader conditions. Exact delta application requires intercepting every storage write: the tensors the loader reports as loaded must match the intercepted copies, and an item without an intercepted copy requires an explicit native skip (Figure A.8). The loader must transform each canonical value independently and deterministically throughout the transition. XOR byte ranges must be disjoint from all other writes, and overlapping overwrites must yield the same bits as a dense refit. NeMo-DCR rejects unsupported loader paths in advance and fails any attempt whose copy breaks these conditions (Table A.2).
6.2 Equivalence to a dense refit
Under these conditions, a NeMo-DCR refit is bit-exact. Let Δv+1 encode \mathcal{E}, and let applyr apply Δv+1 at required receiver r, starting from its storage Prv.
Proposition 1 (Dense-refit equivalence).
If a first attempt passes every check before its commit and the assumptions of Table A.3 hold, including fixed candidate weights Wv+1, committed baseline, source ownership, conversion rules, and loader settings until the commit, then
\operatorname{bits}\!\left(\operatorname{apply}_{r}(\Delta^{v+1},P_{r}^{v})\right)=\operatorname{bits}\!\left(L_{r}(W^{v+1})\right)\kern 5.0pt\forall r, where bits reads all rank-local parameter and buffer storage but excludes request state such as the KV cache, which each transition invalidates.
Proof sketch. XOR reconstructs each new value from the resident version-v bits (Figure A.13, top). Overwrite supplies the value itself. Every changed destination has a record, and destinations without a record keep their version-v bits, which already match a dense refit because their canonical elements did not change. The bottom of Figure 6 outlines the argument, and Appendix A gives the full statement and proof.
6.3 Overwrite recovery and joint commit
Overwrite retries extend Proposition 1 to transitions whose first attempt fails. Partial writes from a failed attempt leave receivers with a mixture of old and updated values. NeMo-DCR stops the failed attempt and waits until its writes can no longer take effect, then recomputes the complete changed set from the same candidate weights and unchanged Bv. Every record becomes an absolute overwrite, supplying the target bits regardless of which earlier writes completed (Figure A.14). The retry thus yields Lr(Wv+1). Receiver membership changes take effect only between attempts, and new or restarted receivers restore the committed version through a dense refit before joining.
At the end of an attempt, the joint commit publishes the updated policy and binds it to its source baseline in a durable, versioned commit record that a control plane holds and changes only by compare-and-set. The coordinator commits after all required receivers have received every expected payload and finished application, device synchronization, and optional post-apply checks. A commit succeeds only if the commit record still equals the version-v record Kv. Owners then update the task trackers and residual baseline in place (Figure B.3). This update finishes before the transition ends or change detection resumes, so the next delta uses the committed baseline.
The commit record also makes coordinator failover safe: a new coordinator reads the same commit record before deciding whether to retry. Appendix B specifies the protocol as Algorithm B.2, with commit deadlines and the control-plane conditions for failover.
7 Pipelined Delivery without a
Cross-Cluster Collective
In ❹, pipelined delivery over object storage or a relay tree overlaps other refit stages without a cross-cluster collective. Owners emit the payloads, and both transports share the refit protocol of Appendix B.
7.1 Object-storage and relay-tree delivery
Payload bytes scale with emitted values, locations, and metadata, but transport topology and retries determine how often payloads cross the cluster boundary. Let Vpayload be the total bytes in unique compressed payloads before replication to receivers, and Vcross the bytes crossing that boundary.
Figure 7: Logical payload movement per attempt in ❹ of Figure 4, with owners O, rollout nodes N, and the object-storage manifest of expected payloads.
Object storage. This transport separates uploading from downloading (Figure 7, left). Owners upload each payload once and list it in a per-attempt manifest of expected payloads. Rollout nodes fetch every listed payload without a direct link. One stored copy in canonical coordinates serves every rollout node, but each download crosses the cluster boundary, so Vcross grows with the number of rollout nodes. A retry uploads the complete changed set again.
Relay tree. Where direct links exist, this transport streams each owner’s payloads to one root in the rollout cluster (Figure 7, right). The root forwards payloads through a balanced tree over local links, and each node’s receivers apply them. Without retransmission, each payload crosses the cluster boundary once per attempt, so Vcross≈Vpayload. Appendix B details both transports.
7.2 Overlapping refit stages
Overlap matters because payload volume scales with the changed set, whereas delta construction cost scales with model size: construction scans source shards and converts required residual tensors. To overlap delta construction and delivery, owners group whole tensors into chunks for comparison and pack the resulting records into buckets, each compressed into one payload (Figure 8, top). Buckets amortize the overhead of metadata, compression, and transfer.
Figure 8: Top: chunks group whole tensors, and buckets pack their records. An unchanged chunk has none (∅) and adds nothing to bucket ℓ, which continues past it. Bottom: stage timeline of residual-task buckets in ❷–❼ of Figure 4 with conversion (conv), comparison (cmp), and the request gate.
In Figure 8 bottom, owners construct bucket ℓ+1 while payload ℓ is sent. After open_apply, receivers in synchronous RL apply ℓ while ℓ+1 arrives. Bounded queues slow upstream stages when receivers fall behind, limiting the number of decoded payloads held in memory. Pipeline startup and drain leave stages idle.
A request gate shields requests from partially applied weights and closes at different times in synchronous and asynchronous RL. In synchronous RL, the coordinator closes it when the first attempt starts, and receivers apply payloads during delivery (Figure 8, bottom). In asynchronous RL, receivers stage the compressed delta in host memory while rollout keeps serving version v. After the last payload arrives, the coordinator closes the gate, and receivers apply the delta in place once current requests finish. The gate reopens when the transition ends.
8 Evaluation
Research questions (RQs) 1–3 and 6 test the four contributions, and each subsection names the contributions it tests. RQs 4–5 test their combined effect.
**RQ1.**How much smaller are XOR masks than overwrite values after compression? (§ 8.2)
**RQ2.**How do direct projection and XOR encoding change construction time and bytes? (§ 8.3)
**RQ3.**Are refits bit-exact, and does training under kills follow dense NCCL? (§ 8.4)
**RQ4.**How much faster is a NeMo-DCR refit than the reference at 120B? (§ 8.5)
**RQ5.**How do latency and speedup change as models grow from 30B to 1T? (§ 8.6)
RQ6. How much of refit latency do relay-tree transport and delta construction occupy? (§ 8.7)
8.1 Experimental setup and bit-exactness
Testbed. The latency testbed has 32 NVIDIA GB300 GPUs on eight training nodes in one AWS region and 64 H100 GPUs on eight rollout nodes in another. The object-storage transport uses Amazon S3. Each node’s cross-cluster flow was measured at up to 5 Gbps. The latency runs use the BF16 Nemotron and Qwen3-235B-A22B checkpoints of Table C.1, with sizes from 63.2 to 2,242 GB, where 1 GB is 109 bytes. The 1T checkpoint doubles the layers of Nemotron-3-Ultra-550B-A55B. We run five warm-up refits and then ten timed refits for each training-side configuration of parallel layout and replica count. The ten timings of each configuration span less than 1 s from minimum to maximum. We average them, then average these means with equal weight.
Refit settings. Both transports use direct projection and the same mixed-mode codec and bounded-queue policy, with the chunk, bucket, and queue sizes of Appendix C. The latency runs use synchronous RL, the training runs use asynchronous RL, and neither uses model-specific logic.
Element change rates. Latency and delta construction runs use 3% or 5% element change rates as stress cases, since both exceed every rate in Figure 2. Changed source elements are chosen uniformly from each owner’s shards and updated with a relative amplitude of about 10−2. Replicated copies receive the same changes.
Timed window. Refit latency covers comparison and projection, residual conversion, encoding and zstd, transport, receiver staging, loader application, device synchronization, ACK, joint commit, and the in-place baseline update (Figure C.5). It excludes benchmark change injection, post-apply checks, and one-time work such as model loading, initial baseline construction, and receiver prewarm.
Reference. The two clusters share no InfiniBand or Elastic Fabric Adapter (EFA) route for NCCL. Of approaches A–D in Table 1, only full-checkpoint transfer meets both G1 and G4. We therefore use a transport-only full-checkpoint reference: each training node uploads an eighth of the checkpoint to shared object storage, and each rollout node then downloads all of it (Figure C.5). The reference excludes checkpoint save and load, to its advantage, whereas NeMo-DCR refit latency includes every stage of the timed window. Speedup is reference time divided by refit latency.
Bit-exactness. Comparisons with dense refits from the same candidate weights confirmed bitwise equality of every parameter and buffer element, including unchanged elements, in the evaluated BF16 receiver configurations. Every measured NeMo-DCR refit also passed its post-apply checks over all written parameter and buffer bits.
8.2 XOR masks are 1.7–2.2× smaller
XOR masks are 1.7–2.2× smaller than overwrite values after zstd level-1 compression (contribution 2, Table 2). We generate 20 million standard-normal values directly in BF16 and add independent Gaussian noise with standard deviation a∈{10−4,10−3,10−2} in FP32. We round each result to the nearest BF16 value, as BF16 training does. The comparison isolates value encoding: XOR and overwrite streams encode the same changed words in input order, without locations or metadata, as Appendix C details.
| Noise standard | Overwrite ↓ | XOR ↓ | Reduction ↑ |
|---|---|---|---|
| deviation a | (MB) | (MB) | |
| 10−4 | 00.74 | 00.33 | 2.21× |
| 10−3 | 07.02 | 03.23 | 2.18× |
| 10−2 | 25.70 | 14.93 | 1.72× |
Table 2: Compressed changed-value sizes for 20 million synthetic BF16 inputs, where 1 MB is 106 bytes.
8.3 Projection saves time, XOR saves bytes
| Qwen3 | Change rate | X (s) | P (s) | F (s) | X (GB) | P=F (GB) |
|---|---|---|---|---|---|---|
| 30B | 3% | 2.96 | 2.77 | 3.20 | 1.49 | 2.44 |
| 30B | 5% | 3.23 | 2.99 | 3.29 | 2.29 | 3.80 |
| 235B | 3% | 20.23 | 18.38 | 19.87 | 11.37 | 18.42 |
| 235B | 5% | 23.07 | 20.42 | 21.96 | 17.58 | 29.36 |
Table 3: Qwen3 delta construction time and payload size before replication. An arrow names the only difference between its modes.
Direct projection makes Qwen3 delta construction 1.08–1.16× faster than full conversion, and XOR encoding cuts payload bytes by 38–40% versus overwrites only (contributions 1 and 2, Table 3). We compare three delta construction modes in the 3% and 5% stress cases on Qwen3-30B-A3B and Qwen3-235B-A22B, which use the same mapping classes. Mode X uses direct projection with mixed XOR/overwrite encoding, P uses direct projection with overwrites only, and F uses full conversion, which assembles and converts full tensors, with overwrites only. P versus F isolates direct projection, and X versus P isolates XOR encoding. Mode X spends 7–13% more construction time than P, yet it is the default because transport dominates relay-tree refit latency (§ 8.7).
8.4 Training follows dense NCCL
under receiver kills
With receivers killed mid-refit, both NeMo-DCR transports follow dense NCCL’s mean reward and KL trajectories (contribution 3, Figure 9). To exercise in-place application and overwrite retries on receivers that survive a kill, we train Qwen3-30B-A3B on agentic tool calls for 50 GRPO steps with the settings of Appendix E. Training and rollout each use four H100 nodes on a second testbed. We compare three methods: a dense NCCL refit that sends every weight, object-storage NeMo-DCR, and relay-tree NeMo-DCR. These runs use real training updates, which change 1.2% of elements per step on average. KL is the estimated per-token KL divergence between rollout and training policies.
Figure 9: Qwen3-30B-A3B reward and per-token KL, averaged over five runs. Mid-refit receiver kills occur only in NeMo-DCR runs.
Every five steps in each NeMo-DCR run, we pick a vLLM instance uniformly at random, kill it mid-refit, and restart it five steps later (Figure E.1). Each kill fails the in-flight attempt, and the overwrite retry repairs any partial writes on the other receivers. Restarted receivers restore the committed version before rejoining. We run each method five times, and each NeMo-DCR run uses independent random choices. In these runs, every NeMo-DCR refit and overwrite retry passed all of its configured post-apply checks. Over the 50 steps, mean reward is 0.41 for dense NCCL and both NeMo-DCR transports.
8.5 15–33× faster refits at 120B
Including delta construction and loader application, 120B refits on both transports take 22.6–49.7 s and are 15.1–33.2× faster than the reference (Table 4). Before replication, compressed payloads including locations total 2.4–3.8% of the 247.2 GB checkpoint, 21–24% less than the uncompressed changed values alone, about 3% or 5% of the checkpoint.
| Method | Change rate | Latency ↓ | Volume ↓ | Speedup ↑ |
|---|---|---|---|---|
| Object storage | 3% | 24.5 s | 5.86 GB | 30.6× |
| Relay tree | 3% | 22.6 s | 5.82 GB | 33.2× |
| Object storage | 5% | 49.7 s | 9.38 GB | 15.1× |
| Relay tree | 5% | 41.6 s | 9.35 GB | 18.0× |
| Reference | N/A | 750 s | 247.2 GB | 1× |
Table 4: Mean 120B refit latency versus the transport-only full-checkpoint reference. Volume is Vpayload for NeMo-DCR and the checkpoint size for the reference, both measured before replication.
8.6 12–40× faster from 30B to 1T
With either transport, refits of 30B–1T models stay 12–40× faster than the reference in the 3% and 5% stress cases, although refit latency grows with checkpoint size, from 8.3–14.5 s at 30B to 150–300 s at 1T (Figure 10). At 5%, the relay tree beats object storage at every size, and its advantage grows from 1.13× at 30B to 1.36× at 1T.
Figure 10: Top: reference time and mean NeMo-DCR refit latency. The band spans both transports and change rates, labels give reference minutes, and speedups sit on the dotted lines. Bottom: mean refit latency of each transport.
8.7 Relay-tree transport dominates latency
Nsight Systems traces show that relay-tree transport dominates refit latency and overlaps delta construction (contribution 4, Figure 11). The construction lower bound covers comparison, projection, residual conversion, encoding, and zstd. The transport lower bound covers intervals from payload submission until the sending endpoint receives the rollout node’s queueing acknowledgment. Each lower bound is the time its intervals cover on the busiest owner or sending endpoint, counting overlaps once (Figure C.5). No transport interval includes the staging or application of its own payload.
Figure 11: Relay-tree ref...
