The Connection Closed Cleanly, So We Saved Half an Answer

A clean socket close is not a finished answer.

A streamed response has three possible endings: completed, truncated, aborted. Most clients collapse all three into one boolean called done. That boolean is exactly where half-answers get saved as final.

This is a debugging retrospective you can replay. I am reconstructing the failure rather than quoting a specific incident, so every step below runs on your own machine.

The symptom: HTTP 200, zero errors, half a sentence

A ticket says the assistant stops mid-sentence. The dashboard disagrees. Error rate is zero, p95 latency is flat, and every response logged status 200.

So which signal lied? None of them. They measured transport health, not answer completeness.

That gap is the whole bug. Your metrics watch the socket, your users watch the sentence.

Step 1 — Record the raw bytes before you theorize

Never debug a stream through a pretty-printer. Save the wire format first, then read it.

mkdir -p fixtures curl -sN 'https://your-provider/v1/chat/stream' \ -H 'Content-Type: application/json' \ -d '{"stream":true,"messages":[{"role":"user","content":"explain the fix"}]}' \ > fixtures/case-001.sse wc -l fixtures/case-001.sse tail -3 fixtures/case-001.sse 

Two things in that tail matter more than everything else. Look for the sentinel data: [DONE], and look at the last finish_reason value.

Ask yourself one question: did the provider tell you why it stopped?

Step 2 — Name the terminal states out loud

Write the states down before touching code. Three states, three different causes, three different handling rules.

Notice what all four rows share. The transport succeeded in three of them. Only one row deserves the word complete.

So why does almost every client return a string instead of a status? Because a string is easier to type, and easier to type is easier to get wrong.

Step 3 — Build a provider that misbehaves on demand

You cannot fix what you cannot trigger. So write a stub provider with three switchable endings.

// stub-provider.mjs — a provider that fails on purpose import express from 'express'; const app = express(); const words = ['The', 'fix', 'is', 'to', 'assert', 'on', 'the', 'sentinel']; const frame = (obj) => `data: ${JSON.stringify(obj)}\n\n`; const delta = (w) => frame({ choices: [{ delta: { content: w + ' ' }, finish_reason: null }] }); const end = (reason) => frame({ choices: [{ delta: {}, finish_reason: reason }] }); app.get('/v1/chat/stream', (req, res) => { const mode = req.query.mode ?? 'ok'; res.setHeader('Content-Type', 'text/event-stream'); res.setHeader('Cache-Control', 'no-cache'); if (mode === 'truncate-length') { words.slice(0, 4).forEach((w) => res.write(delta(w))); res.write(end('length')); // provider says why it stopped res.write('data: [DONE]\n\n'); return res.end(); } if (mode === 'cut-socket') { words.slice(0, 4).forEach((w) => res.write(delta(w))); return req.socket.destroy(); // proxy timeout: no sentinel at all } words.forEach((w) => res.write(delta(w))); res.write(end('stop')); res.write('data: [DONE]\n\n'); res.end(); }); app.listen(8787, () => console.log('stub provider on :8787')); 

npm i express node stub-provider.mjs & curl -sN 'http://localhost:8787/v1/chat/stream?mode=truncate-length' 

You now have reproducibility without a vendor account. That single step kills most stream bugs.

Step 4 — Fix the client, including the bug hiding behind the bug

Here is the shape of the broken client I keep meeting in code review.

// buggy-client.mjs — every ending becomes success export async function readStreamBuggy(url) { const res = await fetch(url); const decoder = new TextDecoder(); let text = ''; for await (const part of res.body) { const s = decoder.decode(part, { stream: true }); for (const line of s.split('\n')) { // bug C: frames split across chunks if (!line.startsWith('data:')) continue; const payload = line.slice(5).trim(); if (payload === '[DONE]') return text; // bug A: sentinel never inspected text += JSON.parse(payload).choices?.[0]?.delta?.content ?? ''; } } return text; // bug B: loop end means success } 

Three bugs, one symptom. Bug A returns before reading finish_reason. Bug B treats an aborted socket as a finished answer. Bug C splits JSON across network chunk boundaries and silently drops fragments.

Note that bug B is invisible in tests that only use happy-path fixtures. That is why step 3 exists.

The fixed version returns a state, not just a string.

// fixed-client.mjs — terminal state is explicit export async function readStream(url, { onDelta } = {}) { const res = await fetch(url); if (!res.ok) throw new Error(`http_${res.status}`); const decoder = new TextDecoder(); let buffer = ''; let text = ''; let finishReason = null; let sawSentinel = false; const handleLine = (line) => { if (!line.startsWith('data:')) return; const payload = line.slice(5).trim(); if (payload === '[DONE]') { sawSentinel = true; return; } const choice = JSON.parse(payload).choices?.[0]; if (!choice) return; if (choice.delta?.content) { text += choice.delta.content; onDelta?.(choice.delta.content); } if (choice.finish_reason) finishReason = choice.finish_reason; }; try { for await (const part of res.body) { buffer += decoder.decode(part, { stream: true }); let idx; while ((idx = buffer.indexOf('\n')) !== -1) { handleLine(buffer.slice(0, idx).replace(/\r$/, '')); buffer = buffer.slice(idx + 1); } } } catch (err) { return { status: 'aborted', text, finishReason, error: err.message }; } if (buffer.length) handleLine(buffer); if (!sawSentinel) return { status: 'aborted', text, finishReason }; if (finishReason !== 'stop') return { status: 'truncated', text, finishReason }; return { status: 'complete', text, finishReason }; } 

The rule that follows is blunt. Persist text only when status === 'complete'. Everything else gets stored as a partial draft with a visible reason.

Step 5 — Freeze the failure as a test

A fix without a test is a rumor. Three cases cover the whole surface.

// stream.test.mjs import test from 'node:test'; import assert from 'node:assert/strict'; import { readStream } from './fixed-client.mjs'; const BASE = process.env.PROVIDER_URL ?? 'http://localhost:8787/v1/chat/stream'; test('clean stream completes', async () => { const r = await readStream(`${BASE}?mode=ok`); assert.equal(r.status, 'complete'); assert.equal(r.finishReason, 'stop'); }); test('max_tokens is not success', async () => { const r = await readStream(`${BASE}?mode=truncate-length`); assert.equal(r.status, 'truncated'); assert.equal(r.finishReason, 'length'); }); test('socket close without sentinel is aborted', async () => { const r = await readStream(`${BASE}?mode=cut-socket`); assert.equal(r.status, 'aborted'); }); 

node --test stream.test.mjs 

Run that in CI against the stub, not against the vendor. Vendor tests are slow, flaky, and expensive to keep green.

Step 6 — Make truncation visible in production

Count endings, not just errors. Emit one counter per terminal state and alert on the ratio.

const counters = { complete: 0, truncated: 0, aborted: 0 }; const record = (status) => { counters[status] += 1; }; const truncatedRatio = () => counters.truncated / Math.max(1, counters.complete + counters.truncated); 

If truncated climbs while complete is flat, your prompts grew and your max tokens did not. That is a product bug wearing a metrics costume.

Where a free tier and a free server actually help

Operator-supplied statement: MonkeyCode offers free model access and a free server option. I have not benchmarked either, and allowances change, so check the current numbers on the project page before planning around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Used honestly, the two pieces cover different steps of this workflow.

1. Replay fixtures cost nothing, so the CI suite stays on recorded bytes.
2. Live sampling needs many cheap calls while you learn which finish_reason values a real provider emits.
3. The free server hosts the stub provider plus a replayer, so a reviewer reproduces your bug in one command.

What it does not give you is a latency guarantee. Never infer production throughput from a shared free environment, and never push load tests at it.

Decision table: fixtures, free tier, or paid path

Who should not use this approach

• You need contractual uptime or latency numbers. Free infrastructure is a sandbox, not an SLA.
• Your prompts carry regulated data. Sanitize before any third-party call, or record nothing.
• Your provider does not use SSE. The sentinel still applies, but the parsing layer does not.
• Nobody owns the observability change. A counter nobody reads changes nothing.

The takeaway I keep relearning

One boolean cannot describe three endings. Model the terminal state explicitly, assert on the protocol sentinel instead of the transport event, and record the bytes before you record your theory.

If you want a cheap place to sample what a real provider actually emits, the free model tier is a reasonable sandbox. Run the harness, print every finish_reason you see, and tell me which one surprised you.