AI & ML
Instructions Before Actions: Prospective and Retrospective Language Supervision for Driving Vision-Language-Action Data
arXiv cs.RO0 views
arXiv:2611.20631v1 Announce Type: cross Abstract: Driving vision-language-action (VLA) models can accept navigation instructions, yet predicting a driver's trajectory does not establish that a model follows the instruction. We study how driving-language supervision is collected and evaluated, distinguishing prospective instructions issued before an action from retrospective labels written after observing it. We develop ADVLAT-Engine to synchronize spoken instructions, road-scene video, and vehicle trajectories, and compare three navigation apps on 48 replayed routes. On 87 maneuvers announced by both Google Maps and a passenger, the passenger uses more scene-dependent language. We evaluate four open driving VLAs before and after fine-tuning on prospective navigation-app drives. Fine-tuning reduces passenger-drive trajectory error by 23-59%, but instruction effects remain inconsistent. At real intersections, counterfactual instructions reveal stronger responsiveness: fine-tuned Alpamayo~2 Super gains 31 and 18 accuracy points at 15 and 5~m, while Qwen-Drive follows alternative maneuvers more reliably with a discrete command than with a sentence. We additionally collected retrospective annotations from four people for the driving recordings and describe controlled comparisons of annotation content, timing, and downstream learning; those analyses are pending. Our findings separate maneuver prediction from instruction-following and establish a framework for comparing prospective and retrospective supervision.
Source:https://arxiv.org/abs/2611.20631PreviousVoxSAOR: Semantic-Aware Occupancy Robustification via Aleatoric Uncertainty ModelingNextSTA: A Split Temporal Adversary Method for Evaluating Robustness of Multimodal RL-based Driving Policies
Related posts

Google’s Gemini 4 Neon tripled its coding score without changing the model
0The New Stack

This GMKtec Ryzen-powered mini PC is a solid daily driver with an $80 discount ahead of Black Friday
0TechRadar

China goes public with stealth UAV manned by crack team of 'self-trained female drone operators' — GJ-11 looks a lot like Northrop Grumman's X-47B now-defunct UCAV
0TechRadar

GTA VI gamers beware — experts warn of a 1300% rise in phishing scams ahead of official launch date
0TechRadar

Siri Modular Apple Watch face gains two new styles with watchOS 27.3
09to5Mac
OpenAI's Coding Agents Now Finish 46% of 2–4 Hour Research Tasks Alone — But the Frontier Is Jagged
0dev.to