AI & ML
Beyond Mimicry: Constrained Exploration with GRPO for Joint Multi-Talker ASR and Diarization under Unknown Speaker Counts
arXiv cs.SD0 views
arXiv:2611.22627v1 Announce Type: new Abstract: Joint multi-talker ASR with speaker attribution and timestamps is particularly challenging when the speaker count is unknown and the output must follow a strict structure. We observe that Supervised Fine-Tuning (SFT)-only alignment of SpeechLLMs is brittle under dense overlaps, often leading to burst hallucinations and malformed speaker/timestamp tags. We propose a unified two-stage generative framework: (i) CoT-augmented SFT that infers the global speaker count before transcription, and (ii) GRPO-based constrained exploration guided by a Multi-dimensional Constraint-Aware Reward (MCAR) that directly optimizes permutation-invariant accuracy while enforcing counting, temporal, and structural constraints. On Libri3Mix and a Dynamic-Mix (2+3) set (speaker count unknown at test time), our model achieves 14.52%/9.24% cpWER and 1.95%/1.12% WDER, respectively, with GRPO yielding a 35%/54% relative cpWER reduction over SFT+CoT.
Source:https://arxiv.org/abs/2611.22627PreviousHigh-Fidelity Binarized Neural VocoderNextBack to Mel: Learning to Bridge Fidelity and Perception in Generative Speech Enhancement
Related posts

Designing a Splash-Zone Ambient Lamp: What IP44 Actually Means in Practice
0dev.to Design
A Canva alternative for Instagram posts, stories and carousels with your brand applied in one click
0dev.to Design
What Might the Next Generation of AI Look Like?
0dev.to Discuss
NVIDIA Q3 FY27 Earnings: $115.8B Revenue, $390B Supply Gamble and the Post-Jensen AI Infrastructure Test
0dev.to Discuss
Do developers actually think about code energy efficiency, or is it just about cloud bills?
0dev.to Discuss
Best AI Humanizer Tools for SEO Articles and Long-Form Blog Posts
0dev.to Productivity