AI & ML
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
arXiv cs.RO0 views
arXiv:2610.39232v2 Announce Type: replace-cross Abstract: Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Source:https://arxiv.org/abs/2610.39232PreviousSV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous DrivingNextMaking Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One
Related posts
Top Dev Tools and Tutorials Trending This Week (2026-11-03)
0dev.to GitHub

DeepSeek Super-Agent คืออะไร? แผนที่เปิดให้อ่าน กับโค้ดที่ตามแผนยังไม่ครบ
0dev.to GitHub
Cockroach Party Shakes India
0dev.to News

O que é OpenAI LifeVal? O benchmark de tarefas do dia a dia
0dev.to News

OpenAI LifeVal Nedir? Günlük Görev Benchmark'ı
0dev.to News
A practical GIF and PNG check for transparent game characters
0dev.to Design