AI & ML
Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMs
arXiv cs.SD0 views
arXiv:2609.31193v2 Announce Type: replace-cross Abstract: Current Audio-Visual LLMs (AVLLMs) struggle with reasoning over videos featuring multi-speaker dialogues. In such videos, resolving "who says what" is crucial, which necessitates trimodal (text-audio-visual) binding. Motivated by these challenges, we systematically investigate how this trimodal binding is achieved in AVLLMs. Specifically, we identify emergent symbolic trimodal binding mechanisms in AVLLMs that utilize modality-specific symbolic variables. By encoding auditory and visual components into symbolic variables-capturing temporal utterance sequences and spatial entity coordinates, respectively-the model establishes cross-modal linking within this abstract space. Crucially, we reveal that when trimodal binding fails, the breakdown predominantly stems from misaligned audio-visual connections. To overcome this bottleneck, we introduce an audio-visual prompting method utilizing an off-the-shelf Active Speaker Detection (ASD) model. By simply overlaying visual bounding boxes on active speakers, this training-free approach yields immediate performance gains across four conversation-centric benchmarks. Moreover, lightweight fine-tuning of fewer than 300 steps on these ASD-prompted-videos extends these gains to three general AV benchmarks, suggesting the generalizability of our method. Our code is available at https://github.com/kaistmm/Who-Says-What.
Source:https://arxiv.org/abs/2609.31193PreviousWhy ML-based cough models do not generalize: a systematic cross-dataset evaluation for tuberculosis screeningNextCollecting Infrastructure-Level Measures for Energy Analysis in Modern Computing Infrastructures
Related posts
What does one finished AI task cost? I measured it.
0dev.to AI
Suno music platform gets new feature for image and video creation
0dev.to AI
How to Trim Audio for a Comedy Show Video — Free, in Your Browser
0dev.to WebDev
AI Internship for Students in Bangalore: Develop Practical Skills and Explore Career Opportunities
0dev.to WebDev
SPF PermError: too many DNS lookups
0dev.to Security

Shipping a 1999 strategy game on Linux ARM64: static linking, SHA-256 installers and 40 new pixel-art portraits (JA2 v1.13)
0dev.to Linux