This is a simplified guide to an AI model called Whisper-Diarization maintained by Thomasmol. If you like these kinds of analysis, you should join AImodels.fyi or follow us on Twitter.
Overview
whisper-diarization combines OpenAI's Whisper Large v3 Turbo speech recognition model with Pyannote 3.3.1 speaker diarization to transcribe audio files while identifying which speaker is talking at each moment. Built by thomasmol, this model delivers word-level and sentence-level timestamps alongside speaker labels, making it particularly useful for conversations, interviews, and multi-speaker recordings. The implementation uses CTranslate2 with faster-whisper 1.1.1 to accelerate Whisper inference while maintaining accuracy. The critical detail before use: this model requires acceptance of Pyannote's user conditions on HuggingFace (specifically the segmentation-3.0 and speaker-diarization-3.1 models) and performs automatic speaker detection by default, though you can specify an exact speaker count if known.
Best use cases
Interview and podcast transcription: Extract clean transcripts from recorded interviews with automatic speaker attribution. Each speaker's segments include timing information, allowing you to build searchable archives where listeners can jump to specific speakers or topics. The model handles natural pauses and overlapping speech better than transcription-only models because diarization context improves word boundary detection.
Meeting and call center analytics: Process recorded calls and meetings to generate searchable transcripts with speaker labels. This enables quality assurance teams to review specific agent interactions, extract actionable insights by speaker role, and maintain compliance records. The automatic speaker count detection saves setup time compared to manual configuration.
Content localization and translation: Combine the built-in translation feature with diarization to translate multi-speaker content while preserving speaker identity. When translate: true, the model converts speech to English transcription, and because diarization runs in parallel, you maintain the speaker labels even after translation. This workflow suits content teams working with multilingual source material.
Subtitle and caption generation: Generate timed speaker labels and text suitable for subtitle files. The word and sentence-level timestamps allow for precise subtitle synchronization, while speaker identification enables caption formatting that distinguishes speakers by name or label. This approach works well for video editing workflows where you need to preserve speaker context.
Training data preparation for speech AI: Extract clean, labeled speech data with timestamps for training custom speech models. The diarization labels allow stratification by speaker, ensuring training sets cover diverse vocal characteristics. The confidence scores (avg_logprob and probability) help filter high-confidence segments for higher-quality training corpora.
Limitations
The model's speaker diarization relies on Pyannote's segmentation and speaker identification models, which struggle with heavy accents, very noisy audio environments, and overlapping speech from multiple speakers simultaneously. In these conditions, speaker boundaries may be incorrectly placed or speaker assignments may be swapped. The automatic speaker detection works well up to roughly 10-15 speakers but degrades with larger speaker counts; you can constrain detection to a specific count (1-50) if you know the speaker composition in advance.
Whisper Large v3 Turbo, while faster than the base model, trades some accuracy for speed. It may miss technical jargon, domain-specific terminology, or proper nouns unless you provide them in the prompt parameter. The prompt mechanism acts as a vocabulary hint, but it is not a guarantee of perfect recognition. Acronyms and loanwords require careful formatting with punctuation to maximize accuracy.
Input constraints are strict: audio must be provided as a Base64-encoded string (file_string), a direct URL (file_url), or uploaded as a file. The model does not specify maximum file duration on the Replicate page, but Whisper models generally degrade with very long files (>1 hour) due to context window limitations. Practical deployments should chunk longer audio or monitor latency for files exceeding 30 minutes.
The model outputs segments with speaker labels and timestamps but does not perform noise reduction, audio enhancement, or stereo channel separation before transcription. Poor audio quality directly impacts both transcription accuracy and diarization quality. If your input has background noise, music, or extreme compression artifacts, consider preprocessing through a dedicated denoiser before submission.
Language detection is automatic if not specified; in multilingual recordings, the model may misidentify language boundaries or code-switch behavior. The translation feature converts to English only; it does not support translation into other languages. Speaker count autodetection respects a maximum of 50 speakers, and attempting to exceed this limit will fail.
The model is licensed under MIT, allowing commercial use, but Pyannote models require separate acceptance of their user conditions. Deployment on your own infrastructure requires a HuggingFace API token inserted into the codebase before building. The model was last updated on 2025-02-19, indicating active maintenance, but it may not reflect the newest Whisper or Pyannote releases.
How it compares
whisperx by daanelson and whisperx by awerks both offer diarization similar to whisper-diarization, but WhisperX historically achieved 70x realtime speed on Whisper Large v2. Choose whisper-diarization if you prioritize Whisper v3 Turbo's improved accuracy and newer training data; choose WhisperX if you need maximum speed or are working with very large audio files where latency matters more than marginal accuracy gains. WhisperX variants may have different diarization backends or tuning, so verify the specific version's performance on your domain before committing.
whisper-large-v3 by nateraw provides transcription without speaker diarization. Use this model if you only need text extraction and speaker identity is irrelevant; use whisper-diarization if you need to know who said what. The plain Whisper model is simpler, faster per token, and uses less memory, but you lose the structured speaker attribution that makes multi-speaker transcripts actionable.
whisper-diarization-advanced by rafaelgalle explicitly includes advanced noise reduction, stereo channel support, and flexible audio preprocessing. Choose whisper-diarization if your audio is clean or you prefer a simpler, faster pipeline; choose whisper-diarization-advanced if you work with call center recordings, noisy meetings, or stereo files where preprocessing can meaningfully improve output quality. The tradeoff is complexity and latency: advanced preprocessing takes time and adds dependencies, but it handles degraded audio better.
speaker-diarization by konieshadow provides diarization without transcription, using Pyannote directly. Use this model if you need only speaker segmentation (labels and timing without text), such as for audio stream monitoring or pre-processing for downstream systems. Use whisper-diarization if you need both transcription and speaker labels in a single integrated call, which is more convenient and ensures alignment between text and speaker segments.
Technical specifications
The model stacks two machine learning pipelines: Whisper Large v3 Turbo (based on OpenAI's Whisper architecture, optimized via CTranslate2 and faster-whisper 1.1.1) and Pyannote Audio 3.3.1 (a PyTorch-based speaker diarization system). Whisper Large v3 Turbo is a large-scale multilingual speech-to-text model trained on 680,000 hours of audio transcripts from the internet; it operates in a zero-shot transfer setting without task-specific fine-tuning. Pyannote speaker diarization uses neural embedding models to cluster speech frames by speaker identity.
Input handling supports three formats: Base64-encoded audio (file_string), direct URLs (file_url), or uploaded file paths (file). The model accepts audio in any format Whisper supports (WAV, MP3, M4A, FLAC, and others). The group_segments parameter (default true) automatically merges speaker segments separated by less than 2 seconds, reducing fragmentation in the output. Speaker count can be manually specified (1-50) or autodetected; language can be specified as an ISO 639-1 code (e.g., 'en', 'es') or autodetected. The prompt parameter accepts a comma-separated list of vocabulary, acronyms, and proper nouns to bias Whisper toward recognizing domain-specific terms. The translate flag (default false) converts output to English even if the source language differs.
Output consists of three fields: segments (a list of dictionaries, each containing speaker label, start/end timestamps, transcribed text, avg_logprob confidence metric, and word-level probability scores), num_speakers (integer, detected or as specified), and language (ISO 639-1 code, detected or as specified). Segment timestamps are in seconds with floating-point precision. Confidence metrics (avg_logprob) indicate overall segment reliability, while probability per word enables downstream filtering or highlighting of uncertain regions.
The Cog containerization framework (version 0.13.7 as of the latest update) packages the model for deployment on Replicate or any machine with a GPU. The model requires HuggingFace authentication via a user token to access Pyannote's gated models. MIT license permits commercial use and redistribution.
Model inputs and outputs
Inputs
- file_string (string): Base64-encoded audio file; mutually exclusive with
file_urlandfile - file_url (string): Direct URL pointing to an audio file; mutually exclusive with
file_stringandfile - file (file path): Uploaded audio file; mutually exclusive with
file_stringandfile_url - num_speakers (integer): Exact number of speakers, or leave empty to autodetect; range 1-50
- language (string): ISO 639-1 language code (e.g., 'en', 'es'); leave empty to autodetect
- translate (boolean): Default
false; iftrue, translates transcription to English - prompt (string): Vocabulary list formatted as comma-separated terms with punctuation; used to bias recognition toward domain-specific words, acronyms, and proper nouns
- group_segments (boolean): Default
true; merges speaker segments separated by less than 2 seconds
Outputs
- segments (array of objects): Each object contains speaker identifier, start time (seconds), end time (seconds), transcribed text, average log probability (
avg_logprob), and word-level probability scores - num_speakers (integer): Detected or specified speaker count
- language (string): Detected or specified ISO 639-1 language code
Getting started
import replicate client = replicate.Replicate() # Transcribe and diarize an audio file from a URL result = client.run( "thomasmol/whisper-diarization", input={ "file_url": "https://example.com/interview.mp3", "language": "en", "num_speakers": 2, "translate": False, "prompt": "Dr. Smith, artificial intelligence, machine learning", "group_segments": True } ) # Print results print(f"Detected language: {result['language']}") print(f"Number of speakers: {result['num_speakers']}") print("\nSegments:") for segment in result['segments']: print(f"Speaker {segment['speaker']}: [{segment['start']:.2f}s - {segment['end']:.2f}s]") print(f" Text: {segment['text']}") print(f" Confidence: {segment['avg_logprob']:.3f}\n") Frequently asked questions
Q: Can I specify the speaker names instead of generic labels?
A: No. The model outputs generic speaker labels (typically "SPEAKER_00", "SPEAKER_01", etc.). You must post-process the output to map speaker labels to actual names using external metadata or manual review.
Q: What audio formats does the model support?
A: The model supports any format Whisper accepts, including WAV, MP3, M4A, and FLAC. Provide audio as a Base64-encoded string, a direct URL, or an uploaded file.
Q: How long can audio files be?
A: The model does not specify a maximum duration on Replicate. Whisper models theoretically handle arbitrary lengths, but practical performance degrades beyond 1 hour due to context limitations and latency expectations. For files longer than 30 minutes, consider chunking and processing separately.
Q: Can I deploy this model on my own infrastructure?
A: Yes. Clone the GitHub repository, accept Pyannote's user conditions on HuggingFace, insert your HuggingFace API token into the code, run cog build, and deploy. You need a machine with a GPU and sufficient VRAM to load Whisper Large v3 Turbo and Pyannote's models simultaneously.
Q: What does the avg_logprob metric mean, and how should I use it?
A: avg_logprob is the average log probability of the Whisper model's predictions for that segment. Higher (less negative) values indicate greater confidence. Use it to filter or flag low-confidence segments for manual review; values below -0.5 often indicate problematic audio or speech characteristics.
Q: Does the model handle languages other than English?
A: Yes. Whisper was trained on 680,000 hours of multilingual audio and supports 99 languages. Leave the language parameter empty to autodetect, or specify an ISO 639-1 code. The translate flag converts any language to English transcription.
Q: Is this model production-ready for commercial applications?
A: Yes, with caveats. The MIT license permits commercial use. The model runs on Replicate with no special setup. However, diarization quality degrades with noisy audio, heavy accents, or many concurrent speakers (>15). For mission-critical applications (legal transcription, medical records), validate accuracy on your specific domain before deployment.
Q: How does prompt engineering improve transcription accuracy?
A: The prompt parameter acts as a vocabulary hint, biasing Whisper toward recognizing specific acronyms, proper nouns, and domain terms. Format terms with punctuation (e.g., "Dr. Smith, C.P.U., machine learning") for best results. This does not guarantee perfect recognition but noticeably improves accuracy on technical vocabulary and rare proper nouns.
