I build SongUp AI, an AI song maker with a set of browser audio tools. One of the most used paid tools is the stem separator: upload a song, get vocals, drums, bass and the rest back as separate tracks. This is the story of how it went from "runs entirely in your browser" to "runs wherever it will actually finish", and the numbers that forced each step.

Version 1: everything on the device

The first version ran a Demucs-family model (htdemucs, exported to ONNX) in the browser with onnxruntime-web. No upload, no server cost, the file never left the computer. On a desktop with 16 GB of RAM it was fine.

The problem showed up in the logs on 15 October: three paying users opened the stem tools on their phones that day and zero finished. The model needs about 2.5 GB of memory while it runs. A phone browser does not have that, and when it runs out the tab is reloaded with no error you can catch. From the user's side the page just blinks and the song is gone.

Version 2: phones go to a server

We had Cloudflare credit, so the server side became a Cloudflare Worker with a Durable Object queue (one job at a time, driven by alarms) and a Container running Node 22, onnxruntime-node and ffmpeg with the same ONNX models baked into the image. Stems land in R2 and are kept for seven days, then a cron deletes them. A D1 table tracks the jobs.

Speed on CPU came out at roughly 1 to 1.5 times the song length: a three-minute song takes two to four minutes. Durable Object alarms have a 15-minute wall limit, so songs are capped at six minutes and 40 MB.

Routing rules in the client:

  • phones, and computers reporting 4 GB or less through navigator.deviceMemory, always use the server;
  • a browser where a run crashed before (we keep a flag in storage) always uses the server;
  • everything else tries the device first and moves to the server once per file on any error, a two-minute stall, or an estimate of more than 12 minutes still to go.

Paid users get 5 server splits a day and 30 a month. Failed jobs do not count.

One cost lesson from this stage: the container stayed awake for three minutes after every job by default. Sixty-three jobs, about 90 minutes of real work, produced hours of billed uptime. sleepAfter = '1m' cut most of it.

Version 3: paid users go to the server everywhere

The device-first rule for desktops looked right until we audited the tools paid users use most. In 14 days the client had reported 108 "stem run cut off (out of memory)" events. Seventy-four of them came from the mashup tool, which splits two songs back to back, and most were on 8 to 32 GB machines running Safari or Firefox, which do not expose deviceMemory at all. The heuristic had no information to work with, so it chose the browser, and the browser died.

So the rule changed: a paid member splits on our servers on every device, in every tool that needs stems (stem separator, mashup, karaoke and acapella, blend). Only when the server has done ten splits for that account in a day does the computer fall back to the browser. Server load at the time was under 30 of the 100 site-wide splits a day we allow, so there was room.

What I would tell someone building the same thing

  1. Memory, not CPU, is the limit for in-browser audio ML. A model that fits on a laptop will take a phone tab down silently. Catch it on the next load (we store a flag before a run and clear it after) because you will not catch it during.
  2. deviceMemory is a hint, not a fact. Safari and Firefox never send it. Any rule built on it needs a default for "unknown", and the safe default is the server.
  3. Audit by user segment, not by average. The average success rate looked fine. Paid users on the tools they paid for did not.
  4. Measure idle time on containers. The job cost was small; the idle after each job was the bill.
  5. Keep both paths. The browser path is still a zero-cost fallback when the server quota is spent.

The tool is at https://www.songupai.com/tools/stem-separator. The stems part needs a paid plan; the models load from our own R2 bucket, nothing is fetched from GitHub at run time.

Not solved yet, in the interest of honesty: Demucs leaves some vocal "ghosting" in the instrumental on dense mixes, the desktop-to-server fallback has not been exercised on a real phone end to end, and our tempo detector is still ambiguous on dancehall (it reads 132 BPM where other tools say 112 or 66). Those are next.

P.S. The song side of the same app, in 28 seconds (a song for a husband, his name in the chorus):