I build Phoenix Editor, a desktop app that takes a long stream or Let's Play recording, cuts the dead air and filler words, and turns the best moments into captioned vertical Shorts. It runs on your own GPU: faster-whisper on NVIDIA, whisper.cpp over Vulkan on AMD and Intel, and the encoder your card has.
For months the test suite ran it against generated fixtures, and everything passed. Then someone called it "vibecoded slop" in public, and I realised I had no evidence they were wrong. Nothing in the repo showed the edits were actually good on real footage.
So I built a harness, ran it on real commentary, and wrote down what came back. This post is the numbers, the bugs they exposed, and the fixes.
The harness
Scoring an edit is harder than it sounds, because "good" is subjective. I picked things I could measure:
- Words lost. Transcribe the source with a separate model (Whisper
medium.en, word timestamps). Any reference word that falls inside a removed range was cut by mistake. - Words clipped. A cut boundary that lands inside a word, so the edit says "basi-" and jumps.
- Fillers removed. Reference "um"s and "uh"s that are actually gone from the render.
- What the Shorts picker chose, checked by watching them.
The test set was CC-BY and CC-BY-SA videos from archive.org and similar, uploaded by the people who made them:
| Length | Kind | |
|---|---|---|
| A raw Twitch stream | 3h 09m | VTuber overlay, horror RPG |
| An FPS first-impressions video | 20m | facecam, lightly edited |
| A two-player strategy game | 11m | two voices |
| Two Let's Play episodes | 50m each | no commentary at all |
The two with no commentary were a free check: the engine should refuse them rather than "edit" a silent video. It did.
Round one
| 3h stream | FPS | strategy | |
|---|---|---|---|
| Removed | 63 of 189 min | 5.7 of 20 min | 2.3 of 11 min |
| Reference words lost | 4.1% | 1.8% | 2.5% |
| Words with a cut inside them | 2.6% | 1.7% | 2.4% |
| Fillers removed | 2 of 19 | 2 of 36 | 3 of 8 |
Losing one word in 25 sounds small until you watch it. Every lost word came from a silence cut. About 60% sat within 0.4 s of a cut edge, so the padding was too tight. The rest were deep inside a "silence" that wasn't: quiet speech under game audio.
The fillers were worse, and the cause was not my code. Whisper doesn't write "um" and "uh" by default. It cleans them out of the transcript, and a filler detector that reads the transcript can't cut what isn't there.
And the Shorts were bad enough to be funny:
- On the 3-hour stream, all five Shorts came from the first eight minutes: pre-game chat about having a cold. None from the two boss fights the stream was named after.
- On the FPS video, three of five were the outro.
- On the strategy video, the intro and the outro.
Why the Shorts picker failed
The first version scored fixed 15-second windows on a 5-second grid by loudness and words per second. That rewards exactly the wrong thing. When does a streamer talk fast and loud? When they're explaining something to the camera: the intro, the outro, the sponsor bit. When something actually happens in the game, they react: "oh", "no no no", "wait", "let's go". Short bursts, not monologue.
And a grid window starts and ends wherever the grid says, which is mid-sentence.
The fixes
Cut edges. Silence cuts now stop short of every transcribed word: 80 ms before it, 120 ms after. One subtle bug on the way: there's a cap on how long a silence cut can be, and it was judged after carving out words. A quiet half hour with one word in it got chopped into dozens of jump cuts, removing 504 s instead of 171. The cap is now judged on the whole pause first.
Fillers. Whisper can be told to write them: hotwords on faster-whisper, an initial prompt on whisper.cpp. That worked, with a catch: primed to hear "uh", it also hears some short real words as "uh". Measured against the reference transcript, the culprits were the low-confidence ones and the implausibly long ones (a 3-second "uh" is a stretched timestamp over real speech). So a filler is only cut when Whisper is at least 40% sure and it's at most 1.2 s long. Leaving an "um" in is cheaper than cutting a word.
There was a second filler bug, and it's my favourite: the fillers that were detected mostly never left the render. The pause padding, scene snapping and word guards that protect speech were also erasing the tiny filler cuts. Only 6 of 27 detected fillers made it out. Filler cuts now skip those steps and keep a 40 ms margin.
Repetition loops. Turning on hotwords together with conditioning on previous text locked faster-whisper into repeating one line from minute 70 of the 3-hour stream to the end: 386 words where there should have been about 1,500. Conditioning is off now, and both backends collapse runs of four or more identical sentences.
Shorts, rebuilt:
- Clips are built from whole sentences, 20 to 45 seconds, so a Short opens on a thought and closes on one.
- The main signal is reaction density, not talk. Loudness and speech rate only count relative to the speaker's own level a few minutes either side, so someone who is loud all night doesn't win everywhere.
- A vision pass labels scenes. Combat and cutscenes add to a clip; menus, loading screens and idle subtract.
- Intros, outros and "links in the description" are ruled out, and the first and last couple of minutes of a long recording are discounted.
- Picks are spread out, with a minimum gap that scales with the recording's length. And if there aren't five good moments, you get fewer Shorts rather than filler.
The layout changed too. On a facecam recording, the old reframer followed the face, so the 9:16 crop was the webcam corner and the gameplay was cropped away. Now the facecam is found (YuNet plus edge refinement) and stacked above the gameplay, with 2 to 3 word captions at the seam.
Round two
| FPS before | FPS after | strategy before | strategy after | |
|---|---|---|---|---|
| Reference words lost | 1.8% | 0.99% | 2.5% | 0.22% |
| Words with a cut inside them | 1.7% | 1.36% | 2.4% | 1.25% |
| Fillers removed from the render | ~2 | 20 of 27 detected | ~3 | 3 of 5 detected |
The Shorts on the FPS video are now five gameplay reactions ("whoa whoa whoa", "it feels like a war zone", "just ate a grenade"). The strategy video gets four mid-game moments.
What's still weak
The raw 3-hour VTuber stream. The avatar has no face to stack, the vision pass mislabels an intermission card as gameplay, and some of the transcribed speech is game dialogue rather than the streamer. Those picks are still poor, and I'd rather say so than pretend.
The filler confidence gate also only works on the NVIDIA path for now, because whisper.cpp's per-word confidence isn't wired in yet.
What I'd tell anyone building something similar
- Test on real footage early. Generated fixtures tested that the code ran. They couldn't tell me it picked the outro.
- Score against an independent transcript. If you score your edit with the same transcript you used to make it, every mistake is invisible.
- Whisper's defaults are tuned for clean text, not editing. If you need disfluencies, ask for them, then filter what comes back.
- "Loud and fast" is not "exciting". Reactions are.
Phoenix Editor runs on Windows and Linux, on AMD or NVIDIA, and nothing uploads. There's a 7-day free trial with every feature and no watermark, and a setup guide if you want to throw a VOD at it. If it picks your outro, tell me.
