Local AI is the most overpromised topic in dev tooling. "Run GPT on your laptop" — sure, if your laptop has 12 GB of VRAM and you enjoy 3 tokens per second.

I've been running local models in production-ish settings for a while (speech recognition, translation, TTS, document analysis). Here is what actually works on consumer hardware, and what doesn't.

Speech recognition: faster-whisper

The best value-per-watt local model I use. CTranslate2 backend, int8 quantization.

  • medium int8 on an RTX 2070: ~4.4 GB of VRAM, a 30-second fragment in 0.36–1.62 s depending on speech density;
  • CPU works too, several times slower — fine for batch transcription, too slow for real-time subtitles;
  • hallucination filter is mandatory: silence and noise produce plausible-looking text that nobody said. A VAD gate before recognition fixes most of it.

Translation

  • Helsinki-NLP OPUS models: ~74 MB, fast, quality is decent for short, well-formed sentences — exactly what subtitles are;
  • NLLB / M2M100: better quality, but check the licence. NLLB is CC BY-NC — fine for personal use, not for a commercial product. I learned that the hard way and swapped it;
  • for anything legal or medical, local NMT is not good enough and you should say so rather than ship it.

TTS

  • Piper: tiny, fast, runs on CPU, sounds robotic but intelligible — good for notifications and accessibility;
  • XTTS: much better quality, needs a GPU and a licence read.

LLMs

A 7–8B model in Q4 quantization takes 8–12 GB of VRAM and produces 20–40 tokens/sec on a decent GPU. That is enough for:

  • summarising long documents;
  • extracting fields from PDFs and invoices;
  • classification and routing;
  • "explain this error" against your own logs.

It is not enough to replace a frontier model for open-ended reasoning — and if you're paying per token, the API is cheaper until you cross a volume threshold.

The decision that actually matters

Local vs API is an economics and privacy decision, not a religious one:

  • API is cheaper at the start — dozens of operations a month don't justify hardware;
  • local is cheaper at volume — thousands of documents a month amortise the GPU in months;
  • local is mandatory for sensitive data — medical, personal, trade secrets;
  • hybrid is often best — sensitive data locally, human-facing drafts through an API.

Start with a prototype on your own data. It tells you both quality and real speed, and the decision becomes arithmetic instead of vibes.

Longer write-up with the hardware table, model sizes and real benchmarks: Local AI on a PC: what actually works. A real product built this way: Stream-Subs.