AI & ML
Great Listeners, Poor Decision-Makers: Rethinking Agentic Audio AI
arXiv cs.SD0 views
arXiv:2611.23847v1 Announce Type: new Abstract: Audio artificial intelligence (AI) now recognizes speech, sounds, and music remarkably well, yet voice agents still act when they should not: they execute commands addressed to others, act on misheard content, and commit before the user finishes speaking. This position article argues that the bottleneck in agentic audio AI has shifted from recognition to decision-making. We propose a reference architecture with an explicit decision layer, separate from perception and reasoning, that chooses whether to act, wait, ask, or remain silent from evidence on addressedness, speaker identity, content confidence, tool reliability, utterance completeness, and action risk. We motivate the design through six recurring decision failures and a decision-theoretic argument that transcript confidence alone is insufficient to justify an action. We implement the layer with fuzzy inference, evolutionary optimization, and sequential decision-making, and evaluate it through simulations, ablations, and recorded audio, including egocentric recordings. Addressedness and speaker evidence matter most, risk-aware gating suppresses costly errors, and well-tuned policies perform similarly, with fuzzy gates more robust to overconfident addressedness estimates. On recorded audio, explicit gating cuts executed side talk from 41\% to 7--10\% while serving 83--85\% of valid requests under successful clarification, outperforming prompting-based baselines. These results support explicit action gating as an architectural requirement for reliable voice agents.
Source:https://arxiv.org/abs/2611.23847PreviousSIGMA: speaker-isolated, symptom-structured multimodal estimation of depression severity from clinical interviewsNextConstructing Dynamic Conversations with Neural Back-channel Generation and Voice-line Scheduling
Related posts
Linkup /research is async. Treating it as synchronous is the bug, in code and in the freeze case
0dev.to
Fuck Your AI Safeguards
0Hacker News
Chrome AI features wasting much bandwidth
0Hacker News
Anthropic hits $2T in Nasdaq debut
0Hacker News

Apple Executive in Charge of iPhone's Camera App Reportedly Retires
0MacRumors
Show HN: Read a Claude Code or Codex session file as a page, in the browser
0Hacker News Show