AI & ML
CoT-WAM: Collaborative Training of Vision-Language Models and World-Action Models via Plan Latents
arXiv cs.RO0 views
arXiv:2611.23803v1 Announce Type: new Abstract: General robot manipulation requires both semantic competence and physical foresight. World action models (WAMs) acquire an understanding of and predictive capability over the physical world by jointly modeling visual foresight and action distributions. However, existing WAMs still lack semantic competence: they have limited ability to perform open-world scene understanding, semantic reasoning, and long-horizon planning. Although vision-language models (VLMs) provide these capabilities, effectively integrating their semantic priors into WAM prediction and control remains challenging. To address this challenge, we propose CoT-WAM, a jointly trained WAM-VLM framework. CoT-WAM uses a learnable embedding to exchange information between the VLM and the WAM, which is supervised by textual and goal-image signals during training to encode rich intent information. A tailored two-stage training strategy enables joint optimization and more effective collaboration between the VLM and WAM. We evaluate CoT-WAM on six benchmarks covering control performance, zero-shot generalization, and long-horizon planning. CoT-WAM achieves SOTA performance on LIBERO, CALVIN, and RoboCasa-Kitchen, and demonstrates stronger zero-shot generalization on LIBERO-Plus and LIBERO-Pro without fine-tuning. On RoboCasa365, it further improves long-horizon planning over baselines with and without external planners. Real-world experiments further demonstrate the effectiveness of our method in practical deployments.
Source:https://arxiv.org/abs/2611.23803PreviousImprove How You Improve: Recursive Policy Self-Improvement for Zero-Shot Open-Vocabulary Multi-Object NavigationNextTacPilot: Does Touch Need to Be Everywhere?
Related posts
ASDL: A Fast Vector Approximation Algorithm for Multiplierless Designs of Convolutional Neural Networks
0arXiv eess.SP
Multi-Modal Environment-Aware Beam Management for Massive MIMO: A Geometry-Driven Virtual Base Station Framework
0arXiv eess.SP
TinyMyo: a Tiny Foundation Model for Flexible EMG Signal Processing at the Edge
0arXiv eess.SP
Chirp-Permuted AFDM: A Versatile and Multifunctional Waveform for 6G
0arXiv eess.SP
Synthesizing hyperspectral Data using generative Models to train spectral Unmixing Methods for low-cost Crop Residue Cover Mapping
0arXiv eess.SP
From Points to Trajectories: Rethinking Multimodal Alignment for Physical Events
0arXiv eess.SP