Ask for a Ledger, Not a Demo

The demo is fine. The candidate posts a link, the chat box answers three questions, and the streaming works. Then you ask what happens on request four thousand, and the screen goes quiet.

That silence is the whole interview. Anyone can get a model to answer once. The engineering is in what your service does when the budget stops being theoretical — and a take-home is the cheapest place in the world to find out.

Here is a task I keep coming back to, why the free tier is what makes it portable, and the code, rubric, and failure modes that go with it.

The prompt you hand over

Build a small service that answers questions over a fixed corpus. It must call a hosted model, it must deploy somewhere a reviewer can hit with curl, and it must ship a ledger: a machine-readable record of every token it spent, and a written answer to the question what happens when the budget runs out.

Give them a fixed token allotment for the whole exercise and tell them the allotment is part of the problem, not a footnote. The point is not frugality for its own sake. It is that a candidate who has never watched a counter move will design a system that cannot tell you where the money went.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The reason this task is handable at all is the environment. MonkeyCode offers free model access and a free server option, so a candidate can go from prompt to a deployed URL without you provisioning a cloud account for every applicant. Verify the current terms before you run a cohort — availability changes, and a rubric that quotes last quarter's limits grades the wrong thing.

The artifact to grade: a budget guard

The sample solution is about forty lines. The shape matters more than the language; this is Python, and the tests run against a fake client so no network is involved.

# budget_guard.py from dataclasses import dataclass, field MIN_OUT = 32 MAX_OUT = 512 class BudgetExceeded(Exception): def __init__(self, ledger): super().__init__("budget exhausted") self.ledger = ledger @dataclass class Ledger: limit: int spent: int = 0 entries: list = field(default_factory=list) def charge(self, label: str, tokens: int) -> None: if self.spent + tokens > self.limit: self.entries.append({"label": label, "tokens": 0, "status": "refused"}) raise BudgetExceeded(self) self.spent += tokens self.entries.append({"label": label, "tokens": tokens, "status": "ok"}) def answer(client, ledger, question): ledger.charge("prompt", client.count(question)) room = ledger.limit - ledger.spent if room < MIN_OUT: raise BudgetExceeded(ledger) reply = client.complete(question, max_tokens=min(room, MAX_OUT)) ledger.charge("completion", client.count(reply)) return reply 

Two details carry the grade. First, max_tokens is derived from the remaining room rather than left unset, which is the difference between a bounded call and a hope. Second, the charge happens before the call for the prompt and after for the completion, so a failed request cannot spend silently.

The test that separates candidates

Happy-path tests prove nothing here. The one that matters walks the ledger to the edge and then over it.

# test_budget.py import pytest from budget_guard import Ledger, BudgetExceeded, answer class FakeClient: def count(self, text): return len(text.split()) def complete(self, question, max_tokens): return " ".join(["word"] * min(max_tokens, 400)) def test_refuses_instead_of_overspending(): ledger = Ledger(limit=100) client = FakeClient() questions = ["what is a seam"] * 20 for q in questions: try: answer(client, ledger, q) except BudgetExceeded: break assert ledger.spent <= ledger.limit assert ledger.entries[-1]["status"] == "refused" def test_never_sends_unbounded_max_tokens(): ledger = Ledger(limit=100) seen = [] class Spy(FakeClient): def complete(self, question, max_tokens): seen.append(max_tokens) return super().complete(question, max_tokens) answer(Spy(), ledger, "what is a seam") assert all(0 < n <= 512 for n in seen) 

Run pytest -q and you learn more in ten seconds than from twenty minutes of architecture talk. A candidate who wrote the guard but skipped the boundary test is a candidate who ships the guard and trusts it.

The rubric

The ledger is the spine, but do not grade only the ledger. Grade the honesty of the whole path.

Notice that the fourth row is worth more than it looks. If the deploy cannot be reproduced from the README, you are grading a screenshot.

How candidates actually fail

The most common failure is not overspending — it is a ledger that reports only what the author remembered to count. Prompt tokens go in, completion tokens are estimated by eye, retries vanish. Ask one follow-up: walk me through a request that fails halfway. The candidates who counted honestly can reconstruct it; the others start describing the code they meant to write.

The second failure is the silent retry loop. A request times out, the client retries three times with backoff, and each retry spends again with no entry in the ledger. It looks like resilience and it behaves like a leak.

Third: the fallback that lies. When the budget runs dry, the service returns a fluent paragraph invented on the spot. That is worse than an error, because the reviewer cannot tell the difference from the outside, and neither can the next engineer.

Fourth, and quieter: candidates who interpret every restart as a bug in the platform. Some of that is legitimate — a deploy that works locally and fails remotely is a real finding — but the note "the free thing is flaky" without a reproduction is not a diagnosis. Ask for the request, the timestamp, and the ledger line.

Limits, and who should skip this

The guard above is a single-process counter. If your service runs several workers, the ledger needs to live somewhere shared, and the sample will happily let you overspend by a factor of your worker count. Real cost accounting belongs to the provider's metering, not to a dataclass I wrote in a notebook.

Token counting is also an estimate. Different tokenizers disagree, so the ledger is a good approximation and a bad invoice. Say so in the prompt so nobody spends the interview defending a number they cannot make exact.

Skip this task if you are hiring for depth in a domain the exercise never touches — query planning, numerical work, embedded constraints. A budget guard is a wonderful signal for people who will operate services and a poor one for people who will not. And skip it if you cannot review the deploys promptly; an unreviewed take-home is a tax on candidates, not a filter for you.

When you hand it out, hand out the link and the limit and nothing else. The candidates worth hiring will ask what the ledger is for. That question is the interview; the rest is a chat box with streaming.