48-Hour Field Notes: My Output File Looked Complete Because I Streamed Straight Into It

A batch job that writes one JSON document per item sounds like the safest code you will write all week, right? No shared state, no transactions, no concurrent writers, just open(), write(), close(). So why did I spend most of two days chasing a downstream consumer that kept failing on files which were, by every check I had, present and readable?

Here is the short version: my writer streamed model output directly into the destination path, and my resume logic treated "file exists" as "file is done". When the process died mid-stream — for whatever reason, and on a small server there are several — I was left with a file that existed, parsed, and lied. This post is what I tried in those 48 hours, what broke, and the harness I would hand to anyone running the same shape of job.

The shape of the job

One worker, one file per item, one model call per file. The call streams structured text, I concatenate the chunks, and I write them out. Nothing clever, which is exactly why I did not budget any engineering time for it.

When I wanted to rehearse the failure on purpose rather than wait for it, I ran the loop against MonkeyCode's free model access on the free server option, because a crash test that costs money per attempt is a crash test you stop running.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Nothing below depends on that choice. Swap in any endpoint, any client, any local stub that yields chunks slowly, and the harness works the same way.

Two failures I could reproduce on demand

I stopped guessing and started killing the process with SIGKILL at different points in the stream. Two distinct bugs fell out, and they compounded each other.

1. Truncated output that still parsed. My format was line-delimited, so a file cut at 60% was still a valid sequence of lines. The consumer read four records instead of six and reported success.
2. Resume logic that skipped the damage. My loop said if the output file exists, skip this job. The broken file existed, so the retry never ran, and the corpus quietly kept the truncated copy forever.

Do you see the pattern? Both bugs are about confusing presence with completion. Neither one is a model problem, and neither one shows up in a happy-path test.

Bucket the failure before you bucket the fix

Before writing code I wrote down where a single output file can be in an inconsistent state, because the fixes are different for each bucket:

• Mid-stream death — the process is gone, bytes on disk are partial, and nothing downstream knows it.
• Multiple writers — a retry starts while the first attempt is still alive, and both write the same path.
• Rename across filesystems — you write a temp file in /tmp and the final path lives on a mounted volume, so the rename is not atomic at all.
• Post-write validation gaps — the bytes are complete but the document is missing a required field.

Bucket one and four are the ones that bit me. Bucket two is the one I had unknowingly created by retrying.

Fix part 1: write to .part, fsync, then rename

The core rule is boring and old: never let the destination path contain a partial document. Write to a sibling file, flush it to disk, and only then do the rename. os.replace maps to rename(2), which is atomic when both paths are on the same filesystem — that is the property you are buying, not speed.

# gen.py — streaming writer that never exposes a partial file from __future__ import annotations import hashlib, json, os, sys from pathlib import Path OUT = Path("out") OUT.mkdir(exist_ok=True) def part_path(job_id: str) -> Path: return OUT / f"{job_id}.json.part" def final_path(job_id: str) -> Path: return OUT / f"{job_id}.json" def stream_to_part(job_id: str, chunks) -> tuple[Path, str, int]: tmp = part_path(job_id) digest, size = hashlib.sha256(), 0 with open(tmp, "wb") as f: for chunk in chunks: # your client yields bytes here  f.write(chunk) digest.update(chunk) size += len(chunk) f.flush() os.fsync(f.fileno()) # bytes are on disk before the rename  return tmp, digest.hexdigest(), size 

Two details matter more than they look. The .part file lives in the same directory as the destination, otherwise the rename can fail with EXDEV and you are back to non-atomic copies. And fsync before replace means a power loss does not leave you with a correctly-named empty file.

Fix part 2: gate the rename on validation

The rename should be the last step of a commit, not the last step of a write. Read the temp file back, parse it, check the fields your consumer actually needs, and only then promote it.

def commit_if_valid(job_id: str) -> bool: tmp = part_path(job_id) if not tmp.exists(): return False payload = tmp.read_bytes() try: doc = json.loads(payload) except json.JSONDecodeError: return False if not doc.get("body") or not doc.get("model_key"): return False os.replace(tmp, final_path(job_id)) return True 

Notice that a failed validation leaves the .part file in place instead of deleting it. That is deliberate: it gives you something to inspect, and it forces you to make an explicit decision on the next run instead of silently re-spending tokens.

Fix part 3: a crash test you can run in twenty seconds

A resume strategy you have not crash-tested is a wish. This test kills the worker mid-stream and asserts the destination never appeared.

# test_crash.py import os, signal, subprocess, sys, time from pathlib import Path final = Path("out/job-1.json") proc = subprocess.Popen([sys.executable, "gen.py", "--job", "job-1", "--slow"]) time.sleep(0.5) os.kill(proc.pid, signal.SIGKILL) proc.wait() assert not final.exists(), "destination appeared before any rename — not atomic" print("ok: destination untouched after SIGKILL") 

Run it in a loop with eight different sleep values and you will find the window where your stream is half-written. Then re-run the worker and confirm that your resume path does the right thing:

python gen.py --job job-1 --slow & sleep 0.5 && kill -9 %1 ls -la out/ # expect job-1.json.part, no job-1.json python gen.py --job job-1 python -c "import json;json.load(open('out/job-1.json'))" 

I am deliberately not pasting a success rate here, because one 48-hour window on one machine is not a benchmark and you would be right to ignore it.

Decision table: what to do with a .part file

For the third row, be honest with yourself: most streaming model clients cannot resume from a byte offset, so "resume" usually means "re-request and re-pay". Deleting the partial file is cheaper than pretending otherwise.

What this does not fix

• Cost of the lost attempt. A crash at 90% of the tokens still costs those tokens on retry; atomic writes protect correctness, not spend.
• EXDEV and network filesystems. Rename atomicity is a local-filesystem property. On some mounted volumes you need a different commit marker entirely.
• Two live writers. os.replace will happily let the second writer win. Guard with an O_EXCL lock file or a single-instance constraint at the platform level.
• Disk exhaustion. fsync can fail with ENOSPC, and a failed commit needs a real error path, not a log line.
• Meaning, not shape. Validation proves the document parses and has the fields you remembered to check. It does not prove the content is correct.

Who should not use this

If you already run your jobs inside a durable workflow engine with activities and retries, you have this behaviour and should not re-implement it. If you are writing thousands of small files per second, per-file fsync will dominate your runtime and you want batching or a log-structured format instead. And if partial output is legitimately useful to your consumer, an append-only log with explicit length framing beats rename-based commits.

What I would repeat, and what I would skip

The part I would repeat is the kill-loop: SIGKILL at eight different offsets, before writing any fix. It converted a vague "sometimes the file is bad" into two named buckets in under an hour. The part I would skip is trusting an existence check as a completion signal, which cost me the rest of the window.

If you want to rehearse a mid-stream crash without burning paid credits on every attempt, MonkeyCode's free model access and free server option are one way to get the loop running — the harness above is the part that matters, whichever endpoint you point it at.

What does your writer do when the process dies between the last chunk and the rename? If the answer is "I am not sure", you know what to test next.