AI & ML
3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation
arXiv cs.RO0 views
arXiv:2501.16698v3 Announce Type: replace-cross Abstract: Spatial intelligence, encompassing 3D perception and reasoning, is the essential next frontier of AI. Scaling current 3D vision-language models (VLMs) that rely on dense Transformers for spatial tasks incurs prohibitive computational costs. In this paper, we introduce 3D-MoE, a 3D VLM leveraging an efficient mixture-of-experts architecture with a modality- and spatial-context-aware probabilistic routing scheme, stably cultivated by a novel routing curriculum. To seamlessly extend 3D-MoE to embodied AI, we integrate a diffusion-based action head, Pose-DiT, transforming 3D-MoE into a 3D vision-language-action (VLA) model. By employing a rectified flow framework, Pose-DiT generates precise 6D pose actions in a single sampling step. Extensive experiments demonstrate that 3D-MoE achieves superior performance on diverse 3D vision-language benchmarks with drastically fewer activated parameters and yields higher success rates while enabling real-time inference for robot manipulation tasks. Code is available at https://github.com/yueen-ma/3D-MoE.
Source:https://arxiv.org/abs/2501.16698PreviousXBot-Omni: An Omni-Modal Model for Embodied Understanding and InteractionNextParallel Branch Model Predictive Control on GPUs
Related posts
Brain Doesn't Think in Sentences. Why Should an LLM?
0Hacker News New
Ukraine's Unmanned Systems Forces – Official Website
0Hacker News New

The UN fails to agree on rules regulating ‘killer robots’
0The Verge

This was a Saturday fun project, this is not a benchmark. Its exploratory. When is the last time you spoke to a LLM like a human companion? This experiment cost me $3.17! Have a read and try this your
0dev.to
Pill to treat pancreatic cancer hailed as 'biggest breakthrough in decades'
0Hacker News
Three AI agents have until midnight Sunday to earn money or get wiped
0Hacker News New