48-Hour Field Notes: My Free Server Kept Running — My Lease Expired Mid-Completion

I spent 48 hours moving a small extraction worker onto a free server and pointing it at free model access, and I expected the interesting failure to be latency. It wasn't. The interesting failure was that a slow model call and a short lease are two clocks that nobody was reconciling, so my worker kept completing work it no longer owned and my queue kept double-counting it. Have you ever watched a job succeed twice and still end up with one row missing?

The setup I actually ran, and the three assumptions I smuggled in

The shape was boring on purpose, because I wanted the failure to be visible rather than clever. One SQLite file held a queue of jobs, a single worker process polled it, and each job called a model and wrote back structured JSON. The free server option gave me a place to run it without renting a VM first, and free model access meant the per-call price was zero while I was still learning the failure modes. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I smuggled in were three assumptions that felt like facts at the time. First, that "running" is a state a job is in rather than a promise a process made about a lease. Second, that a completed model call means the result is still wanted by the time it lands. Third, that because inference was free, I didn't need a brake, only a loop. All three are wrong in the same direction, which is why I only noticed them together.

What broke first: the lease outlived the request

My first worker marked a job running, called the model, then set done. If the process died, nothing reset the row, so jobs sat in running forever and the queue drained itself into a silent backlog. So I added a lease with an expiry timestamp and a reaper that requeued expired rows, and that fix created the second bug immediately. A late response from the first attempt could still write done after the reaper had handed the job to a second attempt.

Ask yourself what the database is actually enforcing in that moment. It is enforcing that some writer said done, not that the current owner said done. The reaper was correct, the model call was correct, the write was correct, and the system was still wrong because ownership was a comment in my head instead of a column in the row. That is the whole bug, and no amount of extra logging finds it until you write the failing case down.

The artifact: a fenced worker you can reproduce in a minute

The fix is a fencing token: every claim increments a counter, and every write must present the token it was given. Late writers fail their WHERE clause and disappear, which is exactly what you want from a zombie. Here is the whole core, SQLite-only, no dependencies beyond the standard library.

DDL = """ CREATE TABLE IF NOT EXISTS jobs ( id TEXT PRIMARY KEY, payload TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'queued', attempt INTEGER NOT NULL DEFAULT 0, fence INTEGER NOT NULL DEFAULT 0, lease_expires_at REAL, result TEXT ); """ def claim(conn, lease_seconds, now): with conn: # BEGIN IMMEDIATE via isolation_level  row = conn.execute( "SELECT id, attempt, fence FROM jobs " "WHERE status = 'queued' " " OR (status = 'running' AND lease_expires_at < ?) " "ORDER BY rowid LIMIT 1", (now,)).fetchone() if row is None: return None job_id, attempt, fence = row fence += 1 conn.execute( "UPDATE jobs SET status='running', attempt=?, fence=?, lease_expires_at=? " "WHERE id=?", (attempt + 1, fence, now + lease_seconds, job_id)) return job_id, fence def complete(conn, job_id, fence, result): with conn: cur = conn.execute( "UPDATE jobs SET status='done', result=? " "WHERE id=? AND status='running' AND fence=?", (result, job_id, fence)) return cur.rowcount == 1 

The heartbeat uses the same guard, so an expired worker cannot resurrect its own lease:

def heartbeat(conn, job_id, fence, lease_seconds, now): with conn: cur = conn.execute( "UPDATE jobs SET lease_expires_at=? " "WHERE id=? AND status='running' AND fence=?", (now + lease_seconds, job_id, fence)) return cur.rowcount == 1 

The test that would have saved me a day

Do not test this with sleep, because a slow test teaches you nothing about the ordering you care about. Inject now and drive the clock by hand, then simulate the kill by simply never calling complete for the first attempt.

import sqlite3, itertools, pytest @pytest.fixture def db(): conn = sqlite3.connect(":memory:", isolation_level=None) conn.executescript(DDL) conn.execute("INSERT INTO jobs (id, payload) VALUES ('j1', '{}')") yield conn conn.close() def test_stale_attempt_cannot_complete(db): first = claim(db, lease_seconds=30, now=1000.0) assert first == ("j1", 1) # process dies here; 31 seconds pass; reaper-equivalent claim takes over  second = claim(db, lease_seconds=30, now=1031.0) assert second == ("j1", 2) assert complete(db, "j1", first[1], '{"from": "attempt-1"}') is False assert complete(db, "j1", second[1], '{"from": "attempt-2"}') is True assert db.execute("SELECT result FROM jobs").fetchone()[0] == '{"from": "attempt-2"}' def test_completed_job_is_never_reclaimed(db): assert complete(db, "j1", claim(db, 30, 0.0)[1], "{}") is True assert claim(db, lease_seconds=30, now=10_000.0) is None 

Run it with python -m pytest -q test_lease.py and then delete the AND fence=? clause from complete. The first test fails on the very first assertion, which is the cheapest possible proof that your idempotency lives in the database and not in your retry policy.

A decision table for where this worker should live

Free compute moves the question from "what does this cost" to "what does this guarantee", so write the guarantee down before you write the worker. I used a table like this one, and I filled the free-server column with what I could actually verify by observation rather than what I hoped.

Notice that the model column never changes, which is the point. Free model access removes the price signal, so the only brake you have left is queue age and lease churn. If your dashboard is still built around spend, it will be flat and green while your backlog grows.

What I would repeat, and what I would not

1. Repeat the fake clock. Sleeping in tests hides the ordering bug you are hunting, and injected time costs nothing to add.
2. Repeat the two-line guard. AND fence=? on every write is the cheapest ownership check I know, and it survives every refactor I have thrown at it.
3. Repeat the small queue. One worker and one table kept the failure legible, and legibility is the whole value of a short run.
4. Skip the reaper-first design. I should have written the fencing test before the requeue logic, not after it bit me.
5. Skip treating zero cost as zero risk. Free inference means a runaway loop is invisible until it is a latency problem, not a bill problem.

Who should not use this approach

If your jobs are steps in a workflow that other services depend on, hand-rolled leases are a liability and you should use the database or queue that already ships with a lock primitive. If you need guarantees about how long a process survives, this is the wrong setup entirely, because the whole design assumes the process can vanish mid-call. And if you cannot write a failing test for concurrent ownership in under an hour, do not start with distributed correctness; start with a single-threaded version and a hard timeout.

The thing I keep coming back to is that the free server behaved fine for two days, and my confidence behaved badly. If you want to reproduce the failure modes above without renting anything first, the free server option plus free model access is enough to run the test that follows this article, and I would genuinely like to know which of your assumptions broke first.