Speaker diarization on a MacBook Air M3: 4.4x faster on the GPU, 2.5x for the whole pipeline

I transcribe my meetings and interviews locally on a MacBook Air M3, with whisper.cpp for speech recognition and pyannote.audio for speaker diarization (working out who speaks when). The first time I ran a real 42-minute meeting through it, the whole thing took 2 hours 9 minutes and loaded the laptop heavily.

This post is about what I found when I looked at what the machine was actually doing, what I changed, and why one stage got 4.4x faster while the whole pipeline got only 2.5x faster. The code and setup guide are on GitHub: ekateglazkova/fast-diarization-apple-silicon.

Why local, and what the pipeline does

My recordings often contain other people's data: employers, numbers, sometimes personal details. I don't want to upload them to a third-party service without every participant's consent. I also wanted one output file in the exact shape I work with next, and a few meetings a week are cheap in the cloud but a steady stream of hours adds up. A hosted service is often faster and more accurate, especially at labelling speakers, so this is a setup that fits my constraints, not a claim that local is better.

I drop an audio file into a folder, answer a dialog about the number of speakers, and a few minutes later get one text file with timestamps and speaker labels:

1. A macOS Folder Action (Automator) watches the folder and starts the script.
2. FFmpeg converts the recording to 16 kHz mono WAV.
3. whisper.cpp (medium model, on the GPU through Metal) produces text with segments.
4. pyannote.audio (speaker-diarization-community-1) works out who speaks when, and the script assigns a speaker to each Whisper segment.

[00:00:12] Speaker 1: What problem are we solving? [00:00:19] Speaker 2: We want to cut transcript waiting time. 

Everything runs on the laptop and no audio is sent anywhere. Only the first installation and the model downloads need the network.

What the machine was doing

I looked at the process list while the pipeline was running. On a 180-second test clip, the diarization process averaged about 343% CPU, which is a bit over three cores' worth, and held about 3.8 GiB of resident memory. In a separate process snapshot, the Folder Actions dispatcher that launches the script showed 97% CPU. That last one is an observation from a single moment, not a measurement. The point is that a stage I expected to be light was heavily loading the CPU.

The cause turned out to be simple. whisper.cpp was already running on the GPU through Metal, but pyannote.audio does not choose a device for you. If you never call .to(...) on the pipeline, PyTorch runs it on the CPU. I did not measure GPU utilization, so I can't tell you how busy the GPU was during diarization. What I can say is that the pipeline was running on the CPU, and the Apple GPU is a supported backend in PyTorch (mps).

To be fair about attribution: the 2 h 9 min figure came from my very first run and is not a controlled baseline. Whisper alone took 9 min 7 s in that run versus 6 min 28 s later, so something else was different too. I therefore compare only runs where I controlled the input, described below.

What I changed

I made two groups of changes. I can show the effect of the set, but I did not measure each change on its own, so I won't claim a number for any single one.

1. How the job is launched. I moved the processing out of the Folder Action into a separate background process started with nohup. The Automator action now finishes immediately instead of waiting for the transcription, and the output goes to a log file:

BASE="$HOME/Documents/Meetings" mkdir -p "$BASE/Transcripts" /usr/bin/nohup /bin/zsh "$BASE/scripts/run_transcription.sh" "$@" </dev/null >> "$BASE/Transcripts/transcription.log" 2>&1 & 

run_transcription.sh puts files in a queue, so several recordings dropped into the folder are processed one after another instead of all at once, and a .processed marker keeps a finished file from being processed again. I tested the queue on real files: the second recording started one second after the first finished. After these changes the dispatcher showed 0% CPU when idle, against 97% in the earlier snapshot.

2. Moving diarization to the GPU. This is the change that matters for speed. In diarize.py the pipeline is moved to the Apple GPU when it is available, and falls back to the CPU otherwise:

device = torch.device("mps" if torch.backends.mps.is_available() else "cpu") pipeline.to(device) 

The script prints which device it used and how long inference took, and calls torch.mps.synchronize() before reading the clock, because MPS operations are asynchronous and without it the timer would stop before the GPU finished.

Results

For a controlled comparison I used one 180-second clip and ran the same pipeline on the CPU and on the GPU. Here RTF (real-time factor) is processing time divided by audio duration, so lower is better.

↓ means the value on the GPU is that many times lower than on the CPU. For time and RTF, lower is better. The whole-pipeline row is an estimate: I added the measured diarization time to the time of the other stages.

Why 2.5x and not 4.4x. Everything except diarization (conversion and Whisper) takes about 26.5 seconds on this clip and did not change. Total time went from 92.8 + 26.5 to 21.3 + 26.5 seconds, so a stage that is 4.4 times faster makes the whole run 2.5 times faster. This is Amdahl's law: the part you didn't touch sets the ceiling.

On a short clip of 26 to 27 seconds the gain was only 1.6x, so the speedup depends on the length of the recording. I haven't broken down why; my guess is that per-run fixed costs matter more when there is little audio, but I didn't test that.

A real meeting. The 42 min 16 s meeting that took 2 h 9 min on my first run now takes 11 min 19 s (RTF 0.27): 4 min 47 s for diarization and 6 min 28 s for Whisper. I am deliberately not dividing 2 h 9 min by 11 min 19 s and calling it an 11x speedup. That first run is not a controlled baseline, as I said above. The 4.4x and 2.5x figures above come from the controlled clip, and those are the numbers I trust.

What I did not measure

• One run per variant. I did not estimate the spread.
• Word error rate (WER) and diarization error rate (DER). I only compared whether the results agreed between runs.
• Energy use and GPU utilization.
• The contribution of each change. The launch setup and the GPU move were made together.
• Alternatives to the medium Whisper model.

Known limits of the scripts: each Whisper segment gets a single speaker, and there is no fallback to the CPU if MPS raises an error in the middle of a computation. The CPU fallback only applies when MPS is not available at all.

The role of AI

I worked with AI assistants. I set the task and the usability criteria, checked the results, and approved every change and every long run. The assistants read logs and code, proposed hypotheses, prepared a minimal patch and backups, and computed the measurements.

Code

The scripts, the benchmark data, and the setup guide for macOS are in ekateglazkova/fast-diarization-apple-silicon. The code is MIT-licensed. The speaker-diarization-community-1 model is under CC BY 4.0 and needs a Hugging Face token to download.

Feedback welcome

This is my first post on dev.to, and I'm just getting started here, so I'd be glad to hear what you think. If you run pyannote on an M1, M2 or M4, I'd like to know whether you see a similar speedup. If you see something I should have measured or measured differently, tell me in the comments.