Why I Stopped Sending My Voice Notes to OpenAI (And Built My Own Engine in Go)

A few months ago, I caught myself doing something stupid.

I was walking down the street, had an idea about a client's server migration, pulled out my phone, and rambled into a popular "AI voice note" app. Ten seconds later, my speech was neatly transcribed into text. Convenient.

Then I paused and thought about what actually just happened under the hood:

1. My phone recorded an uncompressed audio sample of my physical voice.
2. The app uploaded that .m4a file to a public cloud API endpoint.
3. The file was queued on someone else's infrastructure, processed through OpenAI's Whisper API, and retained according to their enterprise data logging window.
4. The transcribed text was beamed back to my phone.

For a shopping list, fine. For unfiltered thoughts, client system architecture, financial notes, or personal venting? That is an absurd privacy posture.

Most modern "AI note apps" are nothing more than thin wrappers over third-party speech APIs. Even when they claim "we don't train models on your data," your raw voice—with your biometric tone, background ambient noise, and unfiltered thoughts—is leaving your perimeter and sitting in commercial cloud buckets.

I decided to stop streaming my voice to OpenAI. Instead, I spent the last few weeks building my own private capture engine using Go, PostgreSQL, and a self-hosted faster-whisper sidecar running on bare Linux.

Here is what I learned about the latency tradeoffs, the memory architecture, and why note-taking systems constantly fail.

The Latency Myth: Cloud APIs vs. Local CTranslate2

The standard argument for using a cloud speech API is speed and compute capacity. The assumption is that a giant cluster in us-east-1 will transcribe audio much faster than a modest virtual server.

In practice, that assumption is dead wrong for short voice notes (10 to 60 seconds).

When you use a cloud API (like OpenAI's v1/audio/transcriptions), you pay three distinct latency taxes:

• Network Upload Latency: Streaming a multi-megabyte audio file from a mobile phone over cellular LTE/5G to an external API gateway.
• Queueing Overhead: Waiting for a spot in a shared multi-tenant cluster.
• TLS Handshake & Gateway Overhead: Establishing connections across multiple corporate boundaries.

Here is what actual measurements look like for a 20-second voice memo:

Running faster-whisper (using CTranslate2 with INT8 quantization on a modern Linux CPU instance) transcribes a 20-second audio clip in roughly 750 to 800 milliseconds.

Because the audio is uploaded directly to my own web server over an existing HTTP/2 or HTTP/3 connection terminated by Caddy, network overhead drops to under 150ms.

Total time from releasing the mic button to text on screen: under 1 second.

Local inferencing is not slower than the cloud. For voice capture, it is twice as fast.

The Stack: Boring, Predictable, and Fast

I wanted zero maintenance overhead. No Kubernetes clusters, no bloated Node microservices, and no multi-gigabyte Docker containers running out of RAM.

┌────────────────────────────────────────────────────────┐ │ Client (Browser / PWA) │ │ Audio recorded via Web MediaRecorder API │ └───────────────────────────┬────────────────────────────┘ │ HTTPS / TLS (Caddy) ▼ ┌────────────────────────────────────────────────────────┐ │ Go Application Server │ │ - Statically linked binary (~17MB RAM) │ │ - Session auth & rate limiting │ │ - Streaming reverse-proxy to local sidecar │ └─────────────┬────────────────────────────┬─────────────┘ │ Local Unix Socket / Port │ SQL ▼ ▼ ┌───────────────────────────────┐ ┌──────────────────────┐ │ faster-whisper Sidecar │ │ PostgreSQL │ │ - Python 3 + CTranslate2 │ │ - Notes, tags, auth │ │ - base.en model in RAM │ │ - Denormalized text │ │ - Audio discarded on output │ │ - Instant indexing │ └───────────────────────────────┘ └──────────────────────┘ 

1. The Web Layer (Go)

The core web app is written in Go. A single compiled binary (srv) that runs as a systemd service, consuming about 14MB of RAM under normal load.

• Handles user authentication (TOTP 2FA, session tokens).
• Serves lightweight server-rendered HTML templates with htmx.
• Manages incoming audio streams and talk-time quotas.

2. The Audio Sidecar (faster-whisper)

Rather than embedding heavy C++ bindings inside the Go process, the transcription worker runs as a dedicated Python sidecar using faster-whisper with the base.en model loaded permanently into memory.

• It exposes a single internal endpoint on loopback (127.0.0.1:8010).
• The Go server forwards the incoming audio stream directly over loopback.
• The sidecar transcribes the audio, returns JSON containing the raw text, and immediately drops the audio buffer from memory.

3. Zero-Retention Memory Hygiene

This was a non-negotiable rule: temporary audio files must never touch a persistent disk.
Many transcription setups write incoming .wav files to /tmp/, run a script against them, and rely on a cron job or cleanup routine to delete them later. That leaves a window where unencrypted voice recordings sit on a hard drive.

In this pipeline:

• The audio is received in memory via an io.Reader.
• Passed directly to the transcription worker via memory streams.
• Discarded the instant the string is emitted.
• If the server loses power, there is zero audio residue anywhere on the filesystem.

The Real Problem With Note Apps: Cognitive Stall

Building the transcription pipeline solved the privacy problem, but it exposed a much bigger issue: how note-taking apps actually work.

Most note systems (Obsidian, Notion, Bear, Apple Notes) make a fundamental mistake: they force you to organize a thought at the exact moment you have it.

You have an idea while walking or stepping out of a meeting. You open an app. It asks you:

• Which folder does this go in?
• Which tags should you apply?
• What should the title be?
• Should you link this to an existing daily note?

By the time you navigate a nested database or pick three tags, the momentum of the thought is gone. This is called cognitive stall. It is why most people end up with dozens of abandoned note systems.

To make voice capture actually work, you have to decouple capture from review:

1. The Capture Phase (Friction < 2 seconds):
   One button. Speak or type. No titles, no mandatory folders, no hierarchy. Raw thoughts land in an unformatted chronologic feed.

2. The Review Phase (The Morning Paper):
   You don't organize notes while taking them. You review them once a day or once a week.

To solve this visually, I built a feature called Free View:

Instead of scrolling through an endless list of cards, clicking one button transforms your raw notes into an editorial morning newspaper. It uses serif typography, multi-column newsprint rules, drop caps, and automatic layout balancing.

Reading your thoughts formatted like a printed front page makes synthesis natural. You read what you thought yesterday with your morning coffee, archive what’s done, and export what matters.

Keeping Data Portable

The final trap of modern SaaS note apps is proprietary database lock-in. If an app stores your thoughts in a custom format or charges you to export them, you are a hostage, not a customer.

Every note in this system has three export rules:

• One-Click Exports: Full archives export any time to standard Markdown (.md), JSON, and plain text.
• Secret Share Links: If you want to show someone a note, the system generates an unguessable 16-hex read-only URL (/s/<slug>) with noindex headers and OpenGraph previews, so it looks like a clean article in Slack or iMessage.
• No Trackers: Zero third-party telemetry, zero analytics scripts, and no marketing tracking cookies.

What I Shipped

I ended up turning this entire architecture into a production app called Tabbynote.

It’s built for anyone who wants instant, voice-first thought capture without handing their raw audio to OpenAI or Google.

• Stack: Go, PostgreSQL, Caddy, faster-whisper (base.en), vanilla CSS.
• Free Plan: 100 notes, 30 minutes of voice transcription a month, Free View layout, and all export formats included. No credit card required.
• Pro Tier: $49/year for unlimited notes, developer theme packs (Catppuccin, Tokyo Night, Dracula), custom monospace typography (JetBrains Mono, Fira Code), and secret share links.

If you want to test how fast self-hosted Whisper feels in a real browser, you can try it at tabbynote.com.

I'd be interested to hear how other developers are handling the privacy tradeoff with local vs. cloud AI in their own tooling. Are you running local inferencing, or is the convenience of OpenAI’s API still worth the data risk for you?