In February I caught myself pasting an internal document into someone else's chat window. Nobody had complained. I had not read a policy that forbade it. I just did not want to keep doing that every day. "I'll host my own. That's a weekend." It was not a weekend. It took seven months. The mines I stepped on along the way are probably the useful part of this post. Everything below is something that actually broke. The first decision: two boxes There is a Mac mini and an NVIDIA DGX Spark on a shelf. The Mac mini runs the application: web, API, PostgreSQL, Redis, and the tool sandboxes. The Spark does inference and nothing else. They talk over a private Tailscale link. There is no deep philosophy here. The Mac mini is cheap to leave on and almost never falls over. The Spark is expensive and occasionally moody. So the moody one got exactly one job. The payoff showed up later: when the Spark is down, the app stays up and says "model unavailable" instead of dying with it. If everything had died together, I think I would have quit somewhere around month three. The main path is vLLM behind a LiteLLM proxy, serving qwen3.8-27b with a 262K context window. External providers (OpenRouter, NVIDIA NIM, Ollama) only enter the picture if you register your own key. Register nothing and every model call stays local. Mistake 1: it was truncating silently This is the one that cost me the most. I assumed 262K was more room than we would ever use. It is not. And when it overflowed, the app quietly dropped the front of the conversation. Here is what that looks like from the outside: the model appears to get stupid. "Did you ignore the file I just gave you?" over and over. I was convinced the problem was the model. I rewrote prompts for two weeks. Now there is an order. Truncate the input. Then reduce max_tokens. If it still does not fit, return HTTP 413 with an audit record and an alert. Loud failure beat quiet failure every single time. Of every design decision in this project, this one changed the most. Mistake 2: giving an agent a shell is fun until it isn't Hand an agent shell, Python, and a browser, and the range of things it can do jumps. It is genuinely delightful. Right up until the day it reads a file you did not intend it to read. Every external MCP server now runs in its own container: --cap-drop ALL, non-root, network policy applied. Off by default. Risky steps wait for a human approval. Then I wired the audit log to alerts. Being able to answer "what did the AI actually do with that file" after the fact is not a nice-to-have. Without it I would not have been willing to hand the thing to my team at all. That was less a technical problem than a nerve problem. Mistake 3: the same number written in two places The app said the upload limit was 100 MB. The reverse proxy disagreed. I spent a full day debugging the wrong layer. Same shape of bug, different corner: vLLM was started with --limit-mm-per-prompt 8, but the app had no per-request image cap of its own. Requests over the limit failed deep in the stack with a spectacularly unhelpful error. Any number you write in two places will eventually disagree with itself. Obvious. Still cost me a day. The first question my team asked When agent tasks went out internally, I got zero questions about features. The first question was "how much did that cost?" So every run now shows turns, elapsed time, and token cost on the card. What the person building it wants to see and what the person using it wants to see are not the same thing. That one stung a little. What we actually use Four things get daily use beyond plain chat. Agent tasks. Give it a goal and it works across multiple turns in a persistent Docker sandbox. When it fails, it reports non-achievement instead of claiming success. That behavior exists because the earlier version did claim success once, and that was worse than failing. Deep research. Decompose, fetch, cross-check, synthesize with citations intact. The pipeline stays visible while it runs, because a spinner with no detail is just anxiety. Discussion mode. Picks two or more specialist agents, gives them the same evidence, runs them in parallel, synthesizes. Off by default. Most questions do not need a panel and I did not want to pay for one every time. Per-role model routing. The model that plans does not have to be the model that judges. agent, judge, research, spawn, review, summary each resolve their own model, local or external. What it is not The desktop app is macOS on Apple Silicon only. The web UI runs anywhere. There is no Kubernetes story and I am not planning one. Single host by design. Setup time is really just "how long does it take you to get a model endpoint running." Ten minutes with Ollama. A weekend or more if you are standing up vLLM from scratch. What I would tell someone starting today Seven months and the list is short. Make failure visible. Degrading quietly is worse than stopping loudly, every time. Give the expensive machine exactly one job. Confining the thing that breaks to one place is how you sleep. Write each number once. The moment it exists in two places, a countdown starts. The first question will be about cost, not features. It's public The source is MIT, so you can review it before you run it. I also keep a weekly development log on the site, including the weeks where things broke. If the log and the code disagree, the code is right and I would like to know. Source: https://github.com/openmake/openmake_llm Live demo: https://chat.openmake.cc (guests get the default local model only) Self-host guide: https://openmake.cc/en/docs/ Dev logs: https://openmake.cc/en/blog/ If you run your own stack, I would like to hear where you tripped.