48-Hour Field Notes: My Resume Saved a Position Instead of a Record, and Nine Inputs Went Missing

A nightly job that rewrites 412 support tickets into short summaries used to run on my laptop, and that was fine until I wanted my laptop back at night. So I moved the batch onto a free server, kept the model calls on free model access, and gave myself 48 hours to decide whether I trusted it. The first full run finished in 31 minutes with exit code 0 and a tidy log line, and nine of the 412 inputs had never been touched at all. Isn't a green build that quietly drops records the worst possible outcome for a batch job?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I moved, and what I refused to move

The split I chose is boring on purpose: the orchestrator, the input files, and the receipts all live on my laptop, and only the worker loop runs on the free server. The worker reads a JSONL input file, calls the model for each record, writes one output line, and exits. I deliberately did not move scheduling, credentials beyond a single scoped key, or anything that needs to survive a reboot.

I am not going to tell you how long the free server keeps a process alive, how much memory it gives you, or what its uptime looks like, because I do not know and I refuse to guess on your behalf. What I do know is that any host I do not control can disappear between two records, so the design has to be correct when the process vanishes. That constraint is the entire reason this article exists.

Hour 0-6: a smoke test that produced a very confident number

My smoke test ran three records end to end and printed processed=3. It also wrote a tiny state file so the real run would not redo those three, which felt tidy at the time. Here is the resume logic I shipped, trimmed down but structurally identical to what ran:

# resume_cursor.py -- the version that skipped nine inputs import json, pathlib STATE = pathlib.Path("state/progress.json") def load_cursor() -> int: if STATE.exists(): return json.loads(STATE.read_text())["cursor"] return 0 def save_cursor(n: int) -> None: STATE.parent.mkdir(parents=True, exist_ok=True) STATE.write_text(json.dumps({"cursor": n})) def run(items, handle): cursor = load_cursor() for i in range(cursor, len(items)): handle(items[i]) save_cursor(i + 1) 

Read that again and ask yourself what i actually refers to. It refers to a position in a list, and a position only means something if the list is built the same way every time. I had written a resume mechanism that stored an integer and hoped the world would stay still.

Hour 6-14: the retry that skipped nine records and repeated two

The first long run died at record 37 when my shell session ended, and progress.json said {"cursor": 37}. Before retrying, I did what felt like housekeeping: I changed the listing call to sorted() so the log would read nicely. That one-line cleanup changed what index 37 pointed at, because the original run had iterated a directory listing in filesystem order.

python -c "import os; print(os.listdir('state/inputs')[:4])" # ['b7.json', 'a1.json', 'c3.json', 'd9.json'] <- filesystem order, no contract python -c "import os; print(sorted(os.listdir('state/inputs'))[:4])" # ['a1.json', 'b7.json', 'c3.json', 'd9.json'] <- the "cleanup" commit 

Os.listdir order is not a promise, and I had built durability on top of an accident. The retry resumed at the new index 37, skipped nine records that had shifted below that boundary, and re-sent two that had shifted above it. Nothing crashed, nothing warned me, and the only visible symptom was a row count I happened to compare by hand.

Hour 14-20: receipts keyed by content instead of a position

The fix is unglamorous: stop remembering where you are and start remembering what you finished. Each record gets a stable job id derived from its input, the prompt version, and the model family, so the same record produces the same id on any host, in any order, on any day.

# receipts.py -- resume by content, not by position import hashlib, json, os, pathlib, tempfile RECEIPTS = pathlib.Path("state/receipts") PROMPT_VERSION = os.environ.get("PROMPT_VERSION", "v1") MODEL_FAMILY = os.environ.get("MODEL_FAMILY", "unset") def job_id(payload: dict) -> str: basis = json.dumps( { "input": payload["input"], "prompt_version": PROMPT_VERSION, "model_family": MODEL_FAMILY, }, sort_keys=True, separators=(",", ":"), ensure_ascii=False, ) return hashlib.sha256(basis.encode("utf-8")).hexdigest()[:32] def receipt_path(jid: str) -> pathlib.Path: return RECEIPTS / jid[:2] / f"{jid}.json" def has_receipt(jid: str) -> bool: return receipt_path(jid).exists() def write_receipt(jid: str, payload: dict, output: str) -> None: path = receipt_path(jid) path.parent.mkdir(parents=True, exist_ok=True) body = { "job_id": jid, "input_sha256": hashlib.sha256(payload["input"].encode()).hexdigest(), "output_sha256": hashlib.sha256(output.encode()).hexdigest(), "prompt_version": PROMPT_VERSION, "model_family": MODEL_FAMILY, "output": output, } fd, tmp = tempfile.mkstemp(dir=path.parent, suffix=".tmp") with os.fdopen(fd, "w", encoding="utf-8") as fh: json.dump(body, fh, ensure_ascii=False) fh.flush() os.fsync(fh.fileno()) os.replace(tmp, path) # atomic rename on the same filesystem 

The driver becomes almost dull, which is the point, and it can now run in any order without losing work:

def run(items, handle): done = skipped = 0 for payload in items: jid = job_id(payload) if has_receipt(jid): skipped += 1 continue write_receipt(jid, payload, handle(payload)) done += 1 print(f"done={done} skipped={skipped}") 

Hour 20-30: reconciliation found nine missing and two duplicated

Before re-running anything, I wrote a reconciliation pass that compares the input set against the receipt store and prints the difference. It does not call a model, it does not need the server, and it takes about two seconds on 412 records.

python - <<'PY' import json, pathlib, sys sys.path.insert(0, ".") from receipts import job_id, has_receipt items = [json.loads(l) for l in open("state/inputs.jsonl", encoding="utf-8")] missing = [job_id(p) for p in items if not has_receipt(job_id(p))] print(f"{len(items)} inputs, {len(missing)} without receipts") pathlib.Path("state/missing.txt").write_text("\n".join(missing)) PY 

That pass reported nine missing receipts, which matched the nine records I had found by hand, and it also surfaced two ids with duplicate output lines further downstream. I spent the next two hours re-running only the missing set instead of the whole batch. Would you rather pay for a 412-record rerun or a 9-record repair?

Hour 30-48: the four checks I now run before calling a batch done

1. Manifest hash. I hash the canonical, sorted input lines and compare the value on my laptop with the value the worker prints at startup, so a reordered or truncated input file is caught before any model call.
2. Missing receipts. The reconciliation script must print 0 without receipts; anything else is a blocker, not a warning.
3. Receipt integrity. For every receipt I check that input_sha256 matches the current input hash, which catches stale receipts written under an old prompt version.
4. Deploy hygiene. The sync command now excludes cursor files and includes the receipt tree, so resume state travels with the code instead of getting lost in a transfer.

rsync -av ./worker/ free-server:/srv/worker/ --exclude 'state/progress.json' rsync -av free-server:/srv/worker/state/receipts/ ./state/receipts/ 

Where this shape fits, and where it does not

Limitations, and who should not use this

Receipts cost you disk and a hash per record, and on tiny inputs the hashing can outweigh the work itself. If your record text embeds a timestamp or a random id, the job id changes on every run and the receipts become noise, so normalize inputs first or do not adopt this. Teams with regulated audit trails need a durable store with retention policy, not files on a shared host, and anyone whose job must emit an exactly-once external side effect needs a downstream dedupe key rather than my pattern alone.

One more honest caveat: this harness protects correctness of my state, and it says nothing about model quality or server reliability. If a call returns something malformed, the receipt still records success, so validate output before you write the receipt and treat a failed validation as no receipt at all.

What I would repeat, and what I would change

I would repeat the reconciliation pass before every rerun, because it turned an anxious guessing game into a two-second fact check. I would also repeat keeping receipts on the laptop, since pulling them back after each run cost me nothing and gave me a local source of truth. I would change the smoke test: it should write receipts and never leave a position marker behind, because a smoke test that lies about progress is worse than no smoke test at all.

If you want to rehearse this shape before you trust it with real volume, the free model access and free server option are enough to run a twenty-record batch twice and watch the second run skip everything with done=0. Just make sure the first thing your smoke test writes is a receipt, not a cursor.