Unified multimodal models stop fighting themselves when the generation objective targets the same continuous representation that understanding already uses and UMM-JEPA does this through masked, coarse-to-fine prediction of those features.

The Real Problem / Why Naive Approaches Fail

The pitch for a unified multimodal model is simple. One backbone reads images, writes images and answers questions about them. You expect the two skills to help each other: a model that can draw a dog should understand dogs better.

That isn't what usually happens. Generative training often adds little or nothing to understanding. Teams that bolt an image-generation objective onto a vision-language model frequently find their VQA and captioning numbers flat or slightly worse, even though the generation samples look good.

The UMM-JEPA abstract gives a plausible diagnosis. The two tasks optimize different representation spaces. Understanding runs on perceptual features, the semantic patch embeddings from encoders like SigLIP or DINOv2. Generation, in most unified designs, optimizes a synthesis-oriented space: discrete VQ codes (Chameleon-style token prediction) or the latents of a VAE that a diffusion head denoises (Transfusion-style). Those spaces exist to reconstruct pixels. They spend capacity on texture, high-frequency detail and exact color and the understanding pathway doesn't need most of it.

So the gradients from generation train the backbone to be good at a representation the understanding head never reads. The two objectives share weights but not targets. At best they coexist. At worst they compete for capacity.

UMM-JEPA's answer, per the abstract, is to reformulate generation as progressive prediction of continuous visual-understanding representations. The model doesn't predict VQ indices or VAE latents. It predicts the features an understanding encoder would produce for the target image, coarse scale first, then finer scales, with masking inside each scale. The name comes from that combination: JEPA-style prediction in representation space, plus masked next-scale structure.

A caveat about scope before going further. The abstract I'm working from is truncated. It gives the motivation and the core reformulation and the title supplies "masked next-scale representation prediction." The concrete mechanics below (pooling scheme, loss, decoder) are my reconstruction from that framing and from the prior work it clearly builds on. They aren't claims about the paper's exact implementation. Check the paper before you treat any of it as canonical.

Architecture & Inner Mechanics

Three lineages meet here and each one solves a different piece of the problem.

JEPA (Joint-Embedding Predictive Architecture). I-JEPA and V-JEPA showed that predicting the representations of masked regions, rather than their pixels, produces features that work well for downstream understanding. The loss lives in an abstract space, so the model isn't penalized for failing to reproduce unpredictable texture. UMM-JEPA keeps that idea and points it at generation.

Next-scale prediction (VAR). VAR replaced raster-order token prediction with coarse-to-fine prediction. It produces a 1×1 map, then 2×2, then 4×4 and so on. Each scale is conditioned on all coarser scales and all tokens within a scale are produced in parallel. Global structure comes first and detail last, which matches how semantic features are organized anyway.

Masked prediction (MaskGIT, MAR). Inside a scale, you don't need to emit every token at once. You can mask a subset, predict it from the visible tokens plus the coarser context and unmask iteratively at inference. MAR also showed that continuous tokens (no vector quantization) work for autoregressive generation if you handle the per-token distribution properly. That point comes back in the gotchas section.

Put together, the training target becomes a pyramid of understanding features. A frozen encoder (or an EMA copy, as in classic JEPA) embeds the image. Its patch grid gets pooled into several resolutions. The unified backbone, conditioned on text, learns to predict masked tokens at scale k given scales 1..k-1 and the visible tokens at scale k.

flowchart TD subgraph Targets["Target construction (no gradient)"] IMG[Training image] --> ENC[Frozen understanding encoder<br/>e.g. SigLIP / DINOv2 ViT] ENC --> PATCH[Patch features 16x16xD] PATCH --> P1[Scale 1: 1x1] PATCH --> P2[Scale 2: 2x2] PATCH --> P3[Scale 3: 4x4] PATCH --> P4[Scale 4: 8x8] PATCH --> P5[Scale 5: 16x16] end subgraph Backbone["Unified backbone (trainable)"] TXT[Text prompt tokens] --> SEQ CTX[Coarser scales 1..k-1<br/>teacher-forced from targets] --> SEQ UP[Upsampled scale k-1<br/>as coarse prior] --> CUR MASK[Mask tokens at random<br/>positions of scale k] --> CUR CUR[Scale k input tokens] --> SEQ[Token sequence] SEQ --> TF[Transformer blocks] TF --> HEAD[Prediction head] end P5 -.target at scale k.-> LOSS HEAD --> LOSS[Loss on masked positions only<br/>in representation space] subgraph Inference["Generation path"] G1[Text] --> G2[Predict scale 1..K<br/>iterative unmasking per scale] G2 --> G3[Predicted feature map] G3 --> G4[Feature-to-pixel decoder] G3 --> G5[Feed directly to understanding head] end

The bottom-right box makes the case for the whole design. The generated output is a feature map in the same space the understanding pathway consumes. Generation gradients shape the backbone's sense of what a semantic feature map of "a red bicycle against a brick wall" looks like and understanding reads from exactly that space. The two tasks share a target.

The cost is that you no longer have pixels at the end. Turning a semantic feature map into an image needs a separate decoder that inverts the encoder's features. It might be a diffusion model conditioned on the features or a learned upsampling decoder. The abstract doesn't say how UMM-JEPA renders pixels, so I'll treat that as an open design slot.

Production Implementation

Below is a self-contained PyTorch reference for the training objective. It's a sketch of the idea and I wrote it to be readable, not to match the paper's code. It runs on CPU with random tensors and on a GPU with real encoder features. To use real data, swap the fake features for the patch tokens of a frozen ViT (with timm or open_clip you can pull forward_features and drop the CLS or register tokens).

# umm_jepa_sketch.py # Reference sketch: masked next-scale prediction of frozen understanding features. # Requires: torch>=2.1 import math import random from typing import List, Sequence import torch import torch.nn as nn import torch.nn.functional as F SCALES: Sequence[int] = (1, 2, 4, 8, 16) @torch.no_grad() def build_target_pyramid(feats: torch.Tensor, grid: int, scales: Sequence[int] = SCALES) -> List[torch.Tensor]: """feats: [B, N, D] patch tokens from a frozen encoder, N == grid * grid. Returns a list of [B, s*s, D] tensors, one per scale, layer-normalized.""" if feats.dim() != 3: raise ValueError(f"expected [B, N, D], got {tuple(feats.shape)}") B, N, D = feats.shape if N != grid * grid: raise ValueError(f"N={N} is not grid^2 with grid={grid}; " "did you forget to strip CLS/register tokens?") x = feats.float().transpose(1, 2).reshape(B, D, grid, grid) pyramid = [] for s in scales: t = x if s == grid else F.adaptive_avg_pool2d(x, s) t = t.flatten(2).transpose(1, 2) # [B, s*s, D] # Per-token normalization keeps the regression target well scaled and # damps the high-norm outlier tokens many ViTs produce. t = F.layer_norm(t, (D,)) pyramid.append(t) return pyramid class NextScalePredictor(nn.Module): def __init__(self, feat_dim: int = 1024, model_dim: int = 768, depth: int = 12, heads: int = 12, text_dim: int = 768, scales: Sequence[int] = SCALES): super().__init__() self.scales = tuple(scales) self.in_proj = nn.Linear(feat_dim, model_dim) self.text_proj = nn.Linear(text_dim, model_dim) self.out_proj = nn.Linear(model_dim, feat_dim) self.mask_token = nn.Parameter(torch.zeros(1, 1, model_dim)) self.pos = nn.ParameterList( [nn.Parameter(torch.randn(1, s * s, model_dim) * 0.02) for s in self.scales] ) self.scale_emb = nn.Embedding(len(self.scales), model_dim) layer = nn.TransformerEncoderLayer( model_dim, heads, 4 * model_dim, dropout=0.0, activation="gelu", batch_first=True, norm_first=True, ) self.blocks = nn.TransformerEncoder(layer, depth) self.norm = nn.LayerNorm(model_dim) nn.init.normal_(self.mask_token, std=0.02) def _embed_scale(self, tokens: torch.Tensor, k: int) -> torch.Tensor: return self.in_proj(tokens) + self.pos[k] + self.scale_emb.weight[k] def _coarse_prior(self, prev: torch.Tensor, k: int) -> torch.Tensor: """Upsample scale k-1 features onto the scale-k grid (VAR-style prior).""" B, n, D = prev.shape s_prev, s_cur = self.scales[k - 1], self.scales[k] g = prev.transpose(1, 2).reshape(B, D, s_prev, s_prev) g = F.interpolate(g, size=(s_cur, s_cur), mode="bilinear", align_corners=False) return g.flatten(2).transpose(1, 2) def forward(self, text: torch.Tensor, pyramid: List[torch.Tensor], k: int, mask_ratio: float): B = text.shape[0] target = pyramid[k] # [B, s*s, D] n_cur = target.shape[1] n_mask = max(1, int(round(mask_ratio * n_cur))) # Random mask per sample at the current scale. noise = torch.rand(B, n_cur, device=target.device) mask = torch.zeros(B, n_cur, dtype=torch.bool, device=target.device) mask.scatter_(1, noise.argsort(dim=1)[:, :n_mask], True) cur = self._embed_scale(target, k) cur = torch.where(mask.unsqueeze(-1), self.mask_token.expand_as(cur), cur) if k > 0: cur = cur + self.in_proj(self._coarse_prior(pyramid[k - 1], k)) parts = [self.text_proj(text)] parts += [self._embed_scale(pyramid[j], j) for j in range(k)] parts.append(cur) seq = torch.cat(parts, dim=1) h = self.norm(self.blocks(seq)) pred = self.out_proj(h[:, -n_cur:]) # current-scale slots only return pred, target, mask def masked_rep_loss(pred, target, mask, cos_weight: float = 0.5): p, t = pred[mask], target[mask] l1 = F.smooth_l1_loss(p, t, beta=1.0) cos = 1.0 - F.cosine_similarity(p, t, dim=-1).mean() return l1 + cos_weight * cos def sample_scale(scales: Sequence[int]) -> int: # Weight by token count so fine scales get gradient proportional to their size, # but keep a floor so the 1x1 and 2x2 scales are never starved. w = [max(s * s, 4) for s in scales] return random.choices(range(len(scales)), weights=w, k=1)[0] def train_step(model, opt, text, enc_feats, grid=16, device="cpu"): model.train() pyramid = [t.to(device) for t in build_target_pyramid(enc_feats, grid)] k = sample_scale(model.scales) mask_ratio = random.uniform(0.5, 1.0) # 1.0 == predict whole scale from coarser context use_amp = device.startswith("cuda") with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=use_amp): pred, target, mask = model(text.to(device), pyramid, k, mask_ratio) loss = masked_rep_loss(pred.float(), target, mask) if not torch.isfinite(loss): opt.zero_grad(set_to_none=True) raise FloatingPointError(f"non-finite loss at scale {k}: {loss.item()}") opt.zero_grad(set_to_none=True) loss.backward() gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step() return loss.item(), k, gnorm.item() if __name__ == "__main__": torch.manual_seed(0) dev = "cuda" if torch.cuda.is_available() else "cpu" model = NextScalePredictor(feat_dim=1024, model_dim=256, depth=4, heads=4, text_dim=768).to(dev) opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.05) for step in range(20): text = torch.randn(8, 32, 768) # stand-in for text encoder states feats = torch.randn(8, 256, 1024) # stand-in for frozen ViT patch tokens loss, k, g = train_step(model, opt, text, feats, device=dev) print(f"step={step:02d} scale={model.scales[k]:>2}x loss={loss:.4f} gnorm={g:.2f}") 

A few design choices in there deserve explanation.

mask_ratio ranges up to 1.0. At 1.0 the model predicts an entire scale from coarser context alone, which is exactly the situation at inference when a new scale begins. If training never shows fully masked scales, the first unmasking step at inference falls outside the training distribution.

The coarse prior is added to every slot at scale k, masked ones included. That way a masked token isn't a blank. It starts from a blurred estimate and only has to predict the residual detail, which is how VAR frames each new scale.

The target pyramid comes from average pooling the finest encoder grid. That's a shortcut. Average-pooled ViT features aren't what the encoder would emit for a downscaled image and they get washed out at 1×1. Running the encoder at several input resolutions gives "honest" multiscale targets at several times the extraction cost. I'd start with pooling and measure whether it matters.

The inference loop follows directly. For each scale, initialize all slots as masked plus the coarse prior, run T refinement passes that commit a growing fraction of tokens, then move up a scale. With deterministic regression you have no per-token confidence to rank by, so commit order is random unless you add a variance head. That limitation leads to the biggest gotcha.

Production Gotchas, Edge Cases & Failure Modes

Regression to the mean kills diversity. The L1/cosine loss above trains the model to predict the conditional expectation of each masked feature. For understanding-style pretraining that's fine and is what I-JEPA does. For generation it's a real problem. Given "a dog on a beach," the expected feature map averages over every plausible dog and a decoder fed that average tends to produce mushy, low-diversity images. MAR handles this for continuous tokens with a small per-token diffusion head that samples from the distribution instead of predicting its mean. Any representation-space generator needs some stochastic mechanism (diffusion head, flow-matching head or explicit noise injection). The abstract doesn't say which one UMM-JEPA uses and I'd look for that first in the full paper.

The frozen encoder sets your ceiling. If the target space is SigLIP features, the model can't generate distinctions SigLIP doesn't encode. Contrastive image-text encoders are known to be weak on exact counts, small text rendering and precise spatial layout. A generator trained in that space inherits those blind spots and so does the decoder.

High-norm outlier tokens. Many large ViTs produce a handful of artifact tokens with huge norms (the "Vision Transformers Need Registers" paper documents this). Without per-token normalization, those tokens dominate an L1 loss. The layer_norm in build_target_pyramid handles this, but it also throws away norm information the understanding head might use. Check that your understanding pathway consumes features under the same normalization you trained generation against. A mismatch there quietly brings back the representation gap the method exists to close.

You still need a pixel decoder. Feature maps don't show up in a demo. Training a decoder from semantic features back to pixels is its own project and semantic features are lossy by design, so reconstructions will drift on fine detail. If your product needs pixel-faithful editing (inpainting a logo, preserving a face's identity), a representation-space generator is the wrong primary tool.

Teacher forcing across scales. In training, coarser scales come from ground truth. At inference they come from the model's own predictions and errors at 2×2 compound into 16×16. The usual mitigations are scheduled sampling or noise injection on the context scales.

Evaluation is two-sided. The whole claim is that generation helps understanding, so a run is only interesting if you measure both: understanding benchmarks against an understanding-only baseline at matched compute, and generation quality through the decoder. Reporting only one side hides exactly the regression this method is supposed to fix.

When I would not use this. If your goal is pure text-to-image quality, a mature latent diffusion stack is the shorter path. If you don't have a strong pretrained understanding encoder for your domain (medical imaging, satellite multispectral), you have no good target space. If your compute budget can't cover both the unified backbone and a separate feature-to-pixel decoder, don't start. The method's value is the coupling between the two tasks. Without a need for both, it's overhead.

Benchmark Comparison & Cost Tradeoffs

I don't have the paper's reported numbers and I won't make some up. What I can give is structural math that applies to any implementation of this design, plus a qualitative comparison of objectives.

| Generation target | Space type | Shares space with understanding? | Needs pixel decoder | Diversity mechanism |
|, -|, -|, -|, -|, -|
| VQ code prediction (Chameleon-style) | Discrete, synthesis-oriented | No | VQ decoder (cheap) | Softmax sampling |
| VAE latent diffusion (Transfusion-style) | Continuous, synthesis-oriented | No | VAE decoder (cheap) | Diffusion |
| Representation alignment (REPA-style) | Diffusion latent plus auxiliary feature loss | Partially | VAE decoder | Diffusion |
| Masked next-scale representation prediction (UMM-JEPA) | Continuous, understanding-oriented | Yes | Feature-to-pixel decoder (expensive) | Must be added (see gotchas) |

Sequential steps at inference. With a 16×16 final grid, raster autoregression needs 256 sequential steps. A five-scale pyramid (1, 2, 4, 8, 16) needs 5 scale steps times T refinement passes per scale. With T = 4 that's 20 forward passes, each over a sequence of at most 32 text tokens + 85 coarser tokens + 256 current tokens = 373 tokens. Fewer, wider passes are GPU-friendly, though the pixel decoder adds its own latency on top.

Target pyramid size. Total tokens across scales: 1 + 4 + 16 + 64 + 256 = 341. At feature dimension 1024 in bf16 that's 341 × 1024 × 2 bytes ≈ 698 KB per image.

| Dataset size | Precomputed targets (bf16, D=1024) | Same in fp8/int8 |
|, -|, -|, -|
| 100K images | ~70 GB | ~35 GB |
| 1M images | ~698 GB | ~349 GB |
| 10M imag...