AI & ML
REAL-2MIX: Evaluating Identity Faithfulness in Audio-Visual Target Speaker Extraction
arXiv eess.AS0 views
arXiv:2608.03964v3 Announce Type: replace Abstract: Audio-visual target speaker extraction should recover the person indicated by the video, rather than repeatedly return the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin benchmark of jointly recorded two-speaker mixtures with synchronized multi-view video and scene-matched solo references. Its 77 scenes contain 7,598 clips; filtering yields 6,038 evaluable mixtures, each queried for both speakers, giving 12,076 paired target queries. We also curate VOXBLINK2-AVSE, comprising 250,828 synchronized audio--lip pairs. A seven-system TF-GridNet study examines visual representations, training corpora, and conditioning strategies. We evaluate transcription accuracy, speaker identity, residual overlap, and perceptual quality. The AVSR early-fusion system achieves 22.99% CER, and 93.38% of its outputs are more similar to the target speaker than to the interferer. Yet only 42.49% satisfy both this identity criterion and the overlap detector's single-speaker criterion; both target queries satisfy these criteria for just 20.77% of mixtures. A separate Real-AVSE challenge study examines development-remix adaptation. The benchmark reveals that favorable content and identity scores need not coincide with low detected overlap, motivating paired-target evaluation.
Source:https://arxiv.org/abs/2608.03964PreviousBridging Spiking and Continuous Dynamics in Spiking Neural Networks: A Muscle-Inspired ModelNextMeasuring Bounded Autonomy for an LLM Decomposition Planner on a Live 5G-Core Testbed
Related posts
What free AI training in Nepal can teach you before you pay for a class
0dev.to

Agents draft. People decide.
0dev.to UX

โค้ดเพิ่ม 400% แต่คุณไม่ได้เร็วขึ้น 400%: GitHub ชี้คอขวดใหม่อยู่ที่การรีวิว
0dev.to GitHub

Drago: Your GitHub Dragon
0dev.to GitHub
Case Study: Automating Tableau to Power BI Migration with Up to 90% Accuracy.
0dev.to Data
Vom Bluttest ins Molekül: Warum die Alamar/Nightingale-Studie räumliches Denken braucht...
0dev.to Data