AI & ML
Assessing Vision-Language Models for Early Prediction of Unsafe Situations in Robotics
arXiv cs.RO0 views
arXiv:2611.24146v1 Announce Type: new Abstract: Vision-Language Models (VLMs) process visual information and textual instructions and, by using their ability to understand and reason about context, are well-suited for predicting robot failures before they occur. However, evidence supporting this remains limited. To address this, we present an empirical evaluation of VLMs for predicting robot failures. In particular, we experiment with Qwen, InternVL, and LLaVA-OneVision across three robotic case studies (HuRoN, Pal Robotics, and CrowdBot), analyzing both video and odometry data for early failure warning (i.e., before an unsafe situation occurs). Specifically, we evaluate their performance in classifying situations as safe, potentially unsafe, and unsafe, together with uncertainty quantified using various metrics, and study the relationship between uncertainty and performance. Our results showed that Qwen provides the strongest overall classification and prediction-reliability performance, while InternVL shows stronger early-warning capability in several configurations. The results further show that the effect of robot-state information and temporal history depends on the model and evaluation objective. Overall, our findings indicate that general-purpose VLMs can support early robot-safety monitoring, but the most suitable configuration depends on whether the priority is accurate assessment of safety-state, reliable confidence and uncertainty, or timely warning before unsafe events.
Source:https://arxiv.org/abs/2611.24146PreviousBiPathCPG: A Novel Load-Sharing Design Architecture for Compound Planetary GearboxesNextBEV-Diffusion: Camera-Only Traversability Navigation with BEV Memory and Efficient Cross-Robot Adaptation
Related posts
AI Agent Observability: What to Log Before Production
0dev.to
AI Agent Evaluation: How to Test Workflows That Change Their Answers
0dev.to
paper-orchestra: A Multi-Agent Pipeline That Turns Experiment Logs Into LaTeX Submissions
0dev.to
Kerberos Through the HTTPS Tunnel: Pre-Auth Capture via the KDC Proxy
0dev.to
We checked 100 well-known sites for AI-crawler readiness: llms.txt, robots.txt, schema, sitemaps
0dev.to
We asked ChatGPT, Perplexity and Gemini which software to buy. Here's who they named first
0dev.to