AI & ML
Seeing and Imagining as One: Unified Video Generation and 4D Perception for Embodied World Modeling
arXiv cs.RO0 views
arXiv:2610.39232v2 Announce Type: replace-cross Abstract: Embodied world models should not only imagine plausible futures, but also perceive the geometric, semantic, and motion structure of the evolving world. We present GenSight, a unified embodied world model that brings future video generation and dense perception into a single pretrained video diffusion transformer. GenSight uses a unified spatiotemporal representation for video, depth, embodiment segmentation, and first-frame-relative 2D flow. Built on a shared DiT backbone, GenSight uses lightweight task modulation to predict all modalities within the same architecture. All tasks are jointly optimized within a unified velocity prediction space. At inference, GenSight supports two complementary modes: multi-step future video generation with aligned dense predictions and single-step dense perception from an observed video. By unifying modality representation, model architecture, and prediction space, GenSight avoids separate modality-specific pipelines and couples appearance, geometry, semantics, and motion in a shared representation. It promotes cross-modal consistency and allows dense perception to benefit from the spatiotemporal priors learned through video generation. Trained on embodied manipulation data spanning multiple robot embodiments and human-hand interactions, GenSight demonstrates strong performance in both video generation and dense 4D perception, highlighting the effectiveness of unifying seeing and imagining for embodied world modeling.
Source:https://arxiv.org/abs/2610.39232PreviousSV-WAM: An Efficient Surround-View World-Action Model for End-to-End Autonomous DrivingNextMaking Kubernetes AI-Ready for NVIDIA Vera Rubin on Day One
Related posts
Why an AI Logo Is Not Done Until It Is a Vector File
0dev.to Design
Descript: The Text-Based Editing Revolution and the Blueprint for Audio-Visual Velocity
0dev.to Productivity

Planning Three Meals a Day at a Phu Quoc Resort
0dev.to Productivity
《What I Learned Building a Pay-Per-Use Voice Cloning Platform》
0dev.to Productivity
Your Europe Christmas trip starts at the visa counter
0dev.to Productivity
Fine-tuned 4B model beats frontier on narrow task (vendor-reported)
0dev.to Machine Learning