AI & ML
Why Multi-Source Audio Fails to Compose: A Geometric Diagnosis and Its Remedy
arXiv cs.SD0 views
arXiv:2611.24410v1 Announce Type: cross Abstract: Multi-source audio-to-image generation exhibits consistent compositional failures when audio tokens occupy geometrically displaced, semantically unanchored positions in the CLIP text-token space, where the diffusion U-Net lacks the prior to reliably attend to them or ground them in distinct spatial regions. We formalize this as the geometric displacement phenomenon and expose it through a two-level clustering analysis comparing BEATs and CLAP embeddings at the raw encoder and projected CLIP token levels. Three geometric properties emerge as important: intra-class compactness, angular alignment with the CLIP text manifold, and inter-token spatial separation. We show that CLAP, whose contrastive language-audio pretraining aligns embeddings with natural language semantics, substantially outperforms BEATs on all three properties, improving the silhouette score by more than 3x and the Audio-Image Content score as well. Building on this analysis, we introduce a multi-token extension that requires no diffusion-backbone retraining and assigns each audio source to a distinct placeholder token in the CLIP tokenizer, complemented by a mixed inference-time objective that combines Attend-and-Excite with an A* spatial separation loss to resolve residual cross-token interference. For evaluation, we construct multi-audio compositional benchmarks from VGGSound (VGG2, VGG3, and VGGN) and extend six standard audio-image metrics to the multi-source setting. Across five benchmarks and four strong baselines, our method achieves strong results on the metrics most directly tied to compositional faithfulness, with AIC improvements of up to 2.25x over the strongest baseline and FID reduced by nearly 2x on two-source generation. We propose geometric displacement as a diagnostic framework for cross-modal generation systems that project learned embeddings into a pretrained contrastive space.
Source:https://arxiv.org/abs/2611.24410PreviousLocation Matters: Layer-wise Prompt Conditioning for Voice--Style Disentanglement in Text-to-SpeechNextPractical Assessment of Agentic Systems in Acoustics
Related posts
Will AI Coding Eat Video Game Engines?
0Hacker News New
Terminal-Bench 4.1
0Hacker News New
Life as a fearful snail: navigating a world of multiple dangers
0Hacker News New
ICE Is Paying AT&T to Dig Through a Mountain of Call Records
0Hacker News New
Ask HN: In one year, what do you think the most unexpected AI event will be?
0Hacker News Ask
Rant: Gemini 4 does not make any sense
0Hacker News Ask