AI & ML
Streaming Generative Video Compression with Embedding Alignment and In-Context Propagation
arXiv eess.IV0 views
arXiv:2611.23352v1 Announce Type: new Abstract: Video compression at ultra-low bitrates presents substantial challenges in preserving perceptual quality. Although pretrained video diffusion transformers (DiTs) provide powerful generative priors, existing generative video compression (GVC) methods built on them typically rely on computationally expensive VAE decoders to reconstruct video frames. Moreover, jointly refining multiple frames to exploit the inherent spatiotemporal modeling capabilities of video DiTs introduces additional buffering delay. To address these challenges, we propose SGVC, a streaming generative video compression framework that adapts a pretrained latent video DiT into a causal pixel-space refiner. Specifically, SGVC projects decoded features from a contextual pixel codec directly into the DiT input embedding space. To bridge the representation gap between codec-derived embeddings and pretrained DiT embeddings, we introduce Embedding Alignment, which aligns the projected embeddings with input embeddings extracted from the original frames during training. The refined DiT hidden states are then mapped directly to the pixel domain through a lightweight reconstruction path, bypassing the pretrained VAE decoder. To further exploit the spatiotemporal priors of the video DiT while maintaining causal decoding, we introduce In-Context Propagation, which reuses layer-wise K/V states across frames to exploit historical generative context, improving temporal consistency without requiring clip-wise buffering. Extensive experiments demonstrate that SGVC achieves competitive rate-perception performance, with average BD-rate savings of 71.97% over S$^2$VC in terms of DISTS. Project Page: https://chigland.github.io/SGVC/
Source:https://arxiv.org/abs/2611.23352PreviousSWIMSS: An Open Multi-Center SwinUNETR Model for Multiple Sclerosis White Matter Lesion SegmentationNextLIC-Audit: How Comparable Are Reported Gains in Learned Image Compression?
Related posts

A bipartisan group of 24+ House lawmakers forms the AI Safety and Innovation Caucus, seeking to balance mitigating AI risks with maintaining US leadership in AI (Owen Dahlkamp/Politico)
0Techmeme

Source: Meta plans to charge $300 for its forthcoming Muse Charm device and include a free 5G data plan from Verizon for up to 12 months (Jyoti Mann/The Information)
0Techmeme
AI Memorized the Internet. That's Not Intelligence. – Yann LeCun [video]
0Hacker News New
Ask HN: Safety Bully Anthropic right steward of our enterprise AI
0Hacker News New
Has ClickHouse Conquered Metrics?
0Hacker News New
An AI agent that pays for every thought and dies when the money runs out
0Hacker News New