AI & ML
CoT-WAM: Collaborative Training of Vision-Language Models and World-Action Models via Plan Latents
arXiv cs.RO0 views
arXiv:2611.23803v1 Announce Type: new Abstract: General robot manipulation requires both semantic competence and physical foresight. World action models (WAMs) acquire an understanding of and predictive capability over the physical world by jointly modeling visual foresight and action distributions. However, existing WAMs still lack semantic competence: they have limited ability to perform open-world scene understanding, semantic reasoning, and long-horizon planning. Although vision-language models (VLMs) provide these capabilities, effectively integrating their semantic priors into WAM prediction and control remains challenging. To address this challenge, we propose CoT-WAM, a jointly trained WAM-VLM framework. CoT-WAM uses a learnable embedding to exchange information between the VLM and the WAM, which is supervised by textual and goal-image signals during training to encode rich intent information. A tailored two-stage training strategy enables joint optimization and more effective collaboration between the VLM and WAM. We evaluate CoT-WAM on six benchmarks covering control performance, zero-shot generalization, and long-horizon planning. CoT-WAM achieves SOTA performance on LIBERO, CALVIN, and RoboCasa-Kitchen, and demonstrates stronger zero-shot generalization on LIBERO-Plus and LIBERO-Pro without fine-tuning. On RoboCasa365, it further improves long-horizon planning over baselines with and without external planners. Real-world experiments further demonstrate the effectiveness of our method in practical deployments.
Source:https://arxiv.org/abs/2611.23803PreviousImprove How You Improve: Recursive Policy Self-Improvement for Zero-Shot Open-Vocabulary Multi-Object NavigationNextTacPilot: Does Touch Need to Be Everywhere?
Related posts
![Physical AI Engineering — Ch.01: Foundations of Physical AI [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwmesn5i4hcar0szww4nk.png)
Physical AI Engineering — Ch.01: Foundations of Physical AI [EN]
0dev.to
587 of OpenAI's 840 Ashby Jobs Report isRemote: true. Twenty-Two Are Remote
0dev.to

Meta now sells 3D movies on Quest VR headsets
0The Verge
![Physical AI Engineering — Ch.10: Medical and Human Centric Physical AI [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvy8fkq2vnw6ud8c0kvic.png)
Physical AI Engineering — Ch.10: Medical and Human Centric Physical AI [EN]
0dev.to
![Physical AI Engineering — Ch.09: Industrial Physical AI [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2ulspenbbtgxps8mti74.png)
Physical AI Engineering — Ch.09: Industrial Physical AI [EN]
0dev.to
![Physical AI Engineering — Ch.08: Autonomous Vehicles and Mobility [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fudcokq8k14p8ppu1mdnf.png)
Physical AI Engineering — Ch.08: Autonomous Vehicles and Mobility [EN]
0dev.to