Kavish is a forward deployed engineer, an engineer who builds AI with a customer, at the customer's side. His customer is an insurer that wants an assistant to read claim files. One firm rule: claim text may never leave the insurer's own cloud account. The pilot cost $1,600 last month. Rosa, from finance, asks: "At ten times the use, can we just run the model ourselves?"
Kavish needs a clear answer, with numbers, by Friday. The answer is not a vibe. It is a sum.
A taxi or a car
Think about how you get around a city. A taxi charges per ride. You pay nothing on days you stay home. A car costs a fixed sum every month, and each extra trip is almost free.
Calling a model through an API is the taxi. You pay per token, a small piece of text, a bit shorter than a word. Self-hosting an open-weight model, one whose learned settings are published so anyone can download and run it, is the car. You rent GPUs, the powerful chips that run models fast, and run the model yourself.
There is also a middle option: a managed service inside your own cloud account. Think of a rental car with a driver. The data stays with you, and someone else keeps the GPUs running.
Left of the break-even point, the taxi is cheaper. Right of it, the car is cheaper. So the real question is: which side are we on?
What owning the car really means
With an API, someone else built and runs everything. If you self-host, you own three layers of work.
The runtime is the model itself, running on a GPU. It sets how fast one copy answers. The infrastructure shares questions between copies, adds copies when busy, and keeps answers steady when traffic doubles. The tooling is releases, watching for slow answers, test questions, and cost per request. It shows problems before customers do.
Owning the car also means on-call: someone must fix it, even at 3 a.m. That person's time is a real cost. A great model on weak infrastructure still gives slow, patchy answers. Ask who owns each layer before you promise a self-hosting project.
The decision tree, in order
Data rules come first. They can decide the answer before cost matters. Kavish's insurer has a hard rule: claim text stays in its own cloud account. That alone rules out a plain API call to a provider.
Then skills: who can run GPUs? Then volume: high and steady favours the car. And "not measured yet" always means start small and measure. The tree is a first filter. It does not know your real prices or quality results.
Do the sums
All numbers here are made up for the example, except where noted. The inputs: API price $2 per million tokens. Pilot use 800 million tokens a month, from last month's bill: $1,600 divided by $2 equals 800. GPUs $4,000 a month for one set. People $2,000 a month for a part-time engineer and on-call time. One set can serve 4,000 million tokens a month.
Pilot on the API: 800 times $2 equals $1,600 a month. Self-hosting: $4,000 plus $2,000 equals $6,000 a month, used or not. Break-even: $6,000 divided by $2 equals 3,000 million tokens a month. Dollars a month divided by dollars per million tokens gives million tokens a month: the use level at which the API bill reaches $6,000. The pilot uses 800 million. That is far below 3,000. So the answer is: stay on the API for now.
Now try ten times the use, 8,000 million tokens. API: 8,000 times $2 equals $16,000 a month. One set of GPUs serves only 4,000 million. GPUs come in whole sets, so you cannot rent 1.5. You need 2 sets: 2 times $4,000 equals $8,000. Add $2,000 for people. Self-hosting is now $10,000 a month. Compare: $16,000 minus $10,000 equals $6,000.
So at ten times the use, self-hosting saves about $6,000 a month on paper. Check quality and on-call first. Note that the $6,000 break-even used one set. The second set moves break-even up, but 8,000 is still above it.
The $2 is made up, but it is close to a real price. As of November 2026, OpenAI's GPT-4o costs $2.50 per million input tokens. On the other side, an NVIDIA H100 rents for about $2.50 an hour on major clouds, roughly $1,800 a month if it runs around the clock. Prices move, so check the current lists. The method does not change: fixed cost divided by the price per million tokens.
The trap to watch
A car parked in the garage costs nothing extra. Idle GPUs still cost money every hour, even at 3 a.m. with no questions to answer. If your use is bursty, a taxi that charges nothing on quiet days can beat a car that charges every hour. And a cheaper API price moves break-even further away: at $0.50 per million tokens, break-even jumps from 3,000 to 12,000 million tokens a month. A cheaper taxi stays better for longer.
Three numbers decide it: your monthly token volume, your API price per million tokens, and your fixed cost, GPUs plus people. If your volume is above break-even, self-hosting is cheaper on paper. The calculator is a first filter, not a quote. It does not know about quality, traffic spikes, or how long setup takes.
For Kavish's insurer, the data rule already points one way, and the sums say the pilot is far below break-even. Stay on the API, or take the middle option, a managed service in the insurer's own account, and measure until the volume tells a different story.
The full interactive chapter, with the decision tree, the break-even calculator, and two customer cases to practise on, is free at https://deployedlab.dev/lessons/32-why-run-models-yourself. 11 chapters are free with no signup, and the full course is $5/month or $39/year at deployedlab.dev.