"Your audio never leaves your machine."

Lots of apps say something like that. YazSes, the offline dictation tool I maintain, says it too. The question I eventually had to answer was: how would I know if it stopped being true?

Not "is it true today". I believed it was. But a project with contributors, optional extras and a year of features doesn't stay the way you remember it. Someone adds a model downloader. Someone adds an update checker. Each one is reasonable, and each one is a hole in a sentence on the front page.

The promise was missing its exceptions

The honest version of "nothing leaves the machine" was never "zero network code". An offline speech app still has to download a model once. It may check for updates if you ask. There's an optional feature that sends dictation to a local LLM for clean-up. There's a remote-dictation mode where you explicitly forward text to another machine over SSH.

So the real promise is: these specific paths exist, they do these specific things, and nothing else talks to the network. That is a list. And a list can be tested.

The list is written down in a design record (ADR-019). Every module that can reach the network is declared and classified:

  • Fetch paths only pull things in: a model, a release, a package index. They never send what you said.
  • Send paths can transmit dictated text. There are exactly two, and both are fenced:
    • the LLM clean-up path, which refuses anything that isn't a loopback address;
    • the remote proxy, which only talks to the host you named on the command line.

Some data may never leave, whatever the user consents to: voiceprints, the local learning corpus, and audio captured from other people in a meeting.

The privacy promise as a list: fetch paths, exactly two send paths, and data that never leaves

The test

tests/test_egress_inventory.py walks every Python file under src/yazses/ with the ast module and collects anything that looks like an outbound primitive:

for node in ast.walk(tree): if isinstance(node, ast.Import): for alias in node.names: if alias.name.split(".")[0] in _NETWORK_ROOTS: hits.add(alias.name) elif isinstance(node, ast.ImportFrom) and node.module: if node.module.split(".")[0] in _NETWORK_ROOTS: hits.add(node.module)

_NETWORK_ROOTS covers the usual suspects: urllib, http, socket, requests, httpx, aiohttp, ftplib, smtplib, asyncio, webbrowser.

Then it compares what it found with what's declared, in both directions:

  • A module that uses the network but isn't declared fails the build. That catches the new downloader nobody registered.
  • A module that's declared but no longer uses the network also fails. That keeps the list from rotting into "things that used to be true".

And one line holds the central claim:

assert len(SEND) == 2, ( "...That is a change to the project's central privacy claim " "and needs an ADR, not a test edit." )

The error message is aimed at the next contributor (or me, six months from now). If you need a third path that can send what the user said, that's a design decision, and the test tells you where to have it.

There's also a test that the scan finds something. A scanner that silently matches nothing is green forever.

It missed one. Then it got wider.

The first version only looked at imports. It missed the SSH tunnel that carries remote dictation, because that module doesn't import socket at all. It runs ssh as a subprocess.

So the scan grew a second pass for spawned programs: tool names inside a list literal, the shape of an argv. Later passes added model loaders that download weights by name, a "open this in the browser" handoff, and code that reuses the shared release downloader. Each one came from asking the same question again: what can this scan still not see?

Today the inventory has more categories than just fetch and send: shell-outs, dependency installers, local-only IPC sockets. The "two paths that can send what you said" number has not changed.

What it can't do

A static scan finds the mechanisms someone thought of. It cannot find a mechanism nobody thought of. A dependency could open a socket from C code; a new library could phone home on import.

So the docs also publish a check you can run yourself, which doesn't trust my scanner at all: run the app in a container with no network.

docker run --rm --network none -v yazses-models:/home/yazses/.cache \ -v "$PWD:/data" ghcr.io/mskazemi/yazses transcribe jfk.wav

That transcribes a file with networking switched off entirely (after the model has been downloaded once into the volume). If it works there, it doesn't need the network. (One related measurement from the ADR: with HF_HUB_OFFLINE=1, a cached Whisper model loads in 1.9 seconds on a blackholed network. Without it, the load was still hanging after three minutes, because the Hugging Face library kept trying to reach the Hub.)

What I took away

  • A promise with unlisted exceptions is not testable. Write the exceptions down first.
  • Compare sets in both directions. "Nothing new appeared" and "nothing old disappeared" are both drift.
  • Test that your detector detects. An empty result is the most dangerous green.
  • Put the reasoning in the assertion message. The person who trips it needs to know why it exists, not just that it failed.
  • Offer users a check that doesn't depend on your code. "Run it with no network" beats "trust my test".

If your app makes a privacy claim, try writing the list. You might find, like I did, that the sentence on your front page had exceptions you'd never written down.

YazSes is offline, open source (Apache-2.0) dictation for Linux, macOS and Windows: github.com/MSKazemi/yazses. The full inventory: ADR-019.

I wrote this with help from an AI assistant for drafting and editing; the design, the test and the opinions are mine, and I checked every fact against the code and the ADR.