Your Free Tier Runs on a Second Clock

A free inference lane is not a cheaper copy of your paid path. It runs on a second clock. Copy your timeouts across and that clock will bite you.

I reviewed a small service that added one free model call into a request handler. The handler had a 1.5 second budget. The new call carried its own 10 second timeout. Nothing crashed. It just quietly doubled p99 and starved the worker pool.

The bug was architectural, not a config typo. Two clocks were running and only one had a ticket.

Two clocks, one ticket

Your clock starts at the edge. Someone above you decided a user will wait 1.5 seconds. The free lane's clock starts whenever capacity shows up. Those two are never synchronized, and no timeout value makes them agree.

So stop passing durations downward. Pass an absolute deadline instead. Every hop reads the ticket and computes whatever it has left.

Durations drift because each layer restarts the count. A 10 second timeout at three layers is a 30 second worst case. Retries multiply it again. An absolute timestamp cannot drift that way.

The data flow

The edge mints exactly one deadline. Everything below receives that object and never mints another.

# deadline.py import time from dataclasses import dataclass class BudgetExhausted(Exception): pass @dataclass(frozen=True) class Deadline: at: float # absolute monotonic timestamp  @classmethod def after(cls, seconds: float) -> "Deadline": return cls(time.monotonic() + seconds) def remaining(self) -> float: return self.at - time.monotonic() def check(self) -> None: if self.remaining() <= 0: raise BudgetExhausted("request budget spent") 

The lane call reads the ticket before it spends anything. If the remaining budget is thinner than your floor, fail fast. Never start work you cannot finish.

# lane.py import asyncio from deadline import BudgetExhausted async def call_lane(fn, deadline, *, floor: float = 0.05): remaining = deadline.remaining() if remaining <= floor: raise BudgetExhausted(f"only {remaining:.3f}s left") return await asyncio.wait_for(fn(), timeout=remaining) 

That floor matters more than it looks. A call with 40 milliseconds left returns a timeout and a wasted connection. Refusing it early lets the caller degrade on its own terms.

Failure domains

The free lane fails differently from your paid path. It can be slow, rate limited, or absent, and all three arrive through the same async call. Your job is to keep that failure inside one box.

Two things leak by default: concurrency and queue time. A slow lane holds a worker slot while it waits. Ten concurrent slow calls become ten idle workers doing nothing for you.

Cap inflight calls per lane, then time the queue separately from the call. You cannot fix a queue you never measured.

# adapter.py (asyncio.timeout needs Python 3.11+; use wait_for on older versions) import asyncio class FreeLane: def __init__(self, client, max_inflight: int = 4): self.client = client self._slot = asyncio.Semaphore(max_inflight) async def ask(self, prompt, deadline): async with self._slot: # queueing here spends the budget silently  async with asyncio.timeout(max(deadline.remaining(), 0.001)): return await self.client.generate(prompt) 

That semaphore line is where budgets quietly die. Waiting for a slot costs wall-clock time and it is not the call. Record it as its own metric or your histogram lies to you.

A test that reproduces the failure

You cannot review a deadline design you never broke. Two tests cover the real failure modes.

# test_deadline.py import asyncio import pytest from deadline import BudgetExhausted, Deadline from lane import call_lane @pytest.mark.asyncio async def test_slow_lane_never_outlives_budget(): async def slow(): await asyncio.sleep(5) return "late" loop = asyncio.get_running_loop() start = loop.time() with pytest.raises((asyncio.TimeoutError, TimeoutError)): await call_lane(slow, Deadline.after(0.2)) assert loop.time() - start < 0.35 @pytest.mark.asyncio async def test_exhausted_ticket_never_reaches_the_network(): async def never_called(): raise AssertionError("should not run") with pytest.raises(BudgetExhausted): await call_lane(never_called, Deadline.after(-1)) 

Run both and read the durations, not just the green checks.

python -m pytest -q test_deadline.py python -m pytest -q --durations=5 

The first test asserts a five second stall cannot outlive a 200 millisecond budget. The second asserts an exhausted ticket never touches the network. If your suite has no equivalent, you are trusting configuration you never tested.

Where MonkeyCode fits

MonkeyCode's free model access is the second clock this harness was built to measure. It gives you a separate lane to profile without spending your production budget on experiments.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The program advertises ten million free tokens and a free server option. Those figures are operator-supplied, so verify the current terms yourself before planning capacity around them. I am deliberately not quoting model names, quotas or uptime here, because I have not measured them.

What matters for the review is placement, not the vendor. Point the adapter at the free lane, keep the deadline object unchanged, and read two histograms. If queue wait dominates, your cap is wrong. If call time dominates, your floor is too low.

The free server option is the same exercise applied to topology. Run the adapter there and ask one question: does a restart lose anything I cannot rebuild? If yes, keep state off it.

What I would change next

Three things, in order.

Put the deadline in the trace context so one request shows one budget across services. Right now each service mints its own ticket and the traces disagree.

Move admission before the semaphore, not after it. Rejecting at the door costs one comparison. Rejecting after the queue costs you the queue.

Split the latency histogram by lane and by outcome. A single p95 across paid and free hides the exact difference you built the lane to observe.

Who should not use this

Do not route regulated data through a lane you do not control. Do not put a free lane on a hard real-time path. And skip all of this if you cannot emit per-request timing, because the design is unverifiable without it.

The architecture lesson survives either way. A tier without its own clock is just a second outage waiting on the same timeout.