The file you generate is a memory budget, not a format: streaming PDFs, images and ZIPs under a hard ceiling

An endpoint that turns a database query into a downloadable file is one of the few places where your code touches every byte it produces. Most handlers do work on a few kilobytes of JSON and forget it. A file generator holds an object whose size is a function of someone else's data — a statement with 1,200 line items, a photo straight off a phone, a folder of receipts — and the platform it runs on has a memory limit that does not care how reasonable your intent was.

The version of this that people ship first looks like this: build the file in memory, set the right content type, return it. It passes the test where one user downloads one small report. Then a customer selects last year instead of last month, six people click at once, and the endpoint returns a Cloudflare 1102 — "Worker exceeded resource limits" — with Exceeded Memory in the dashboard. On a good day. On a bad day the response has already started, the headers already said 200, and the recipient gets a file that opens to nothing.

This post is about the second category of problem: generating large files — PDFs, images, ZIPs, big CSVs — in a runtime whose memory ceiling is hard, shared, and a hundred and twenty-eight megabytes. The numbers below come from a local harness I ran while writing this (CPython 3.12, reported peak RSS per case, each case in its own process). Platform limits are quoted from Cloudflare's documentation and marked as such. Where I could not measure something, I say so instead of rounding it into a fact.

The ceiling you are budgeting against

From Cloudflare's Workers limits page: memory is 128 MB per isolate, on both the Free and Paid plans, and it includes the JavaScript heap and WebAssembly allocations. The same page says something more important than the number:

This limit is per-isolate, not per-invocation. A single isolate can handle many concurrent requests.

That single sentence changes the arithmetic of every file endpoint. Your per-request allocation is not compared to 128 MB. It is compared to 128 MB divided by however many requests that isolate is serving at that instant — plus whatever the isolate already paid to start up. The docs also describe what happens at the boundary: when an isolate exceeds the limit, in-flight requests are allowed to complete and subsequent requests go to a new isolate, and under extreme load the runtime may cancel incoming requests outright. So the failure is not a clean per-request error. It is a capacity cliff with a retry storm behind it, because the natural client response to a failed download is to try again.

Two more documented facts matter for the design space:

• Response body size has no enforced limit, and there is no hard duration limit for HTTP-triggered Workers — as long as the client stays connected, the Worker can keep streaming. That is the license to stream.
• CPU time is 10 ms on Free and 5 minutes on Paid (default 30 s), which is a different budget from memory and fails with the same 1102. A generator that streams slowly and computes expensively can be killed by the clock while its memory looks perfect.

The runtime even names the exact failure you get from buffering: the error Memory limit would be exceeded before EOF appears, per the docs, "when attempting to buffer a response body that exceeds the limit". That message is a diagnosis. If you see it, the answer is not a bigger machine; the answer is that you are holding the whole body.

The equation

Write the budget down before writing the generator:

peak_isolate = fixed_cost + Σ over concurrent requests (live_bytes_of_that_request) 

fixed_cost is everything the isolate pays regardless of this request: the runtime, your bundle's module-level state, and — the one that surprises people — any font or lookup table you load at module scope. live_bytes is not the size of the file. It is the size of everything the file touches while it exists: decoded pixels, per-page buffers, compression windows, strings, copies, base64 expansions, and the request body you read instead of streaming.

Three levers follow directly, and they are the whole article:

1. Cut the fixed cost so the variable part has room.
2. Make live bytes independent of output size — that is what streaming means, precisely.
3. Cap concurrency in the process, so that N× live bytes still fits.

Most outages I have seen in this shape came from violating (2) while patching (3), or from never looking at (1).

Fixed cost first: the font you never counted

Before blaming the file, measure what an empty request costs. I registered a CJK font and drew a single line of Korean text with reportlab:

A 12.9 KB PDF cost 68 MB of memory. Nothing in the output hints at that, and nothing in the output is wrong — the cost is the font loaded into the process so that glyphs can be outlined. More than half of the isolate's ceiling went to a font, before any content existed.

Where that memory lands depends on a line of code that looks like a detail, so it is worth being precise. If the font is registered once at module scope, it is paid once per isolate and then sits there: the isolate starts life with roughly 60 MB left for every concurrent request, which is a smaller number than most people plan against. If the font is read or parsed per request — a path handed to a library, a readFileSync inside the handler, a fresh Font object per document — you pay it again for each concurrent request, and two of those nearly-empty PDFs are already at 136 MB. Same font, same code, two very different failure curves, and only one of them shows up in a test with one user.

The fix is not in the request path. It is a build step:

Read the last row carefully, because it is the trap: subsetting is itself a memory-hungry operation, and doing it per request at 20,000 glyphs peaked at 161.7 MB — over the ceiling before the PDF even started. Subsetting belongs in CI, producing one artifact per language, and the request path should only ever load that artifact. If your text is genuinely open-ended (user-supplied names in arbitrary scripts), you have three honest options: ship a small set of pre-subset ranges and fall back for the rest, move generation off the request path to a queue consumer or container, or generate to storage and hand out a URL.

The same shape shows up outside fonts: an embedded Unicode table, a timezone database, a template engine compiled at import, a zod schema (Cloudflare's own guidance specifically recommends zod 4.5.0 or later, because earlier versions used substantially more memory per schema). All of it is fixed_cost, and all of it is invisible in a profiler that only looks at the request handler.

Peak tracks live bytes, not output bytes

Here is the measurement that motivates streaming, and it is worth internalising because the ratio is boringly consistent. I wrote a minimal PDF writer twice: once collecting every page's content and joining at the end, once emitting each page as it is produced. Both wrote into the same non-seekable sink.

The baseline for an interpreter with this script loaded is about 9 MB.

• Naive: peak ≈ baseline + 1.95 × output. Joining creates the string and the copy that receives it; a 20 MB file wants 40 MB of room, and the peak is reached at the end, which is the worst possible moment — all the work is done, and the response has not started.
• Streamed: peak ≈ baseline + 0.5 KB per page. The 20,000-page case wrote 54.6 MB and never exceeded 19.8 MB, and what grew there was the offset table and the cross-reference block, not the pages.

Two honest footnotes. First, the streamed case is not perfectly flat: about 0.5 KB per page accumulates because a PDF must remember one offset per object, and my toy writer also builds one flat /Kids array proportional to page count (a real writer uses a balanced page tree; the growth is the same order, and it is why 20,000 pages is a different engineering problem from 200). Second, the library you reach for may not stream at all. The same test with reportlab's canvas — a mature, capable library — showed one write to the sink for the entire document, because the canvas holds the document until save():

Peak scaled with pages while the output stayed under 4.1 MB. The lesson is not that the library is bad; it is that "I passed a stream to it" is not the same as "it streams", and the only way to know is to count writes and watch RSS. Count the write calls. If the count is one, you are buffering, and your stream is decoration.

Three shapes, and when each is right

A fourth shape is easy to forget and often the correct one: generate once, cache the artifact. If the same report is requested by five people in a morning, generation is a scheduled job and the endpoint is a GET of an object. This is the only shape where a 5-minute CPU job is acceptable, because there is no waiting client and no request memory.

The rule I use: if the output size depends on a row count, stream it. If the output size is bounded and small (an invoice for one order, a QR image, a 400-line CSV), buffering is fine, and the buffering version is worth keeping for its simplicity. The mistake is choosing buffering because it is simpler and never checking whether the bound is real.

PDF is a gift to streaming writers

PDF looks like the worst case for streaming and is actually one of the best, because of where its index lives: the cross-reference table is at the end of the file. A writer that emits pages in order only has to remember integers, and startxref at the tail tells the reader where those integers are.

Three constraints to respect:

1. /Length must precede the stream it describes. A content stream object is << /Length N >> stream … endstream, and N is the byte count of what follows. You cannot write the header before you know the size, so you have to buffer one page — which is exactly the right amount. If your library writes an indirect length (/Length 12 0 R) and fills that object afterwards, you can stream at finer grain, but per-page buffering already makes memory independent of document size.

2. The tail is the largest synchronous object you build. My 20,000-page run emitted one 800,093-byte write for the cross-reference block. That is fine, but it is a real allocation, and if your page count can reach six digits it is worth writing xref entries in batches.

3. Subset fonts and downsample images at build time, not per request — see the numbers above. A 300-DPI photo printed at 40 mm wide carries roughly 470 pixels of width; embedding 4,000 is paying 70× the pixels for nothing.

A sketch of the shape, in a Worker, with the size known up front via FixedLengthStream (more on that below):

// Pages are produced one at a time; only offsets are retained. export default { async fetch(req: Request, env: Env): Promise<Response> { const orderId = new URL(req.url).searchParams.get("order")!; const pages = await env.DB.prepare( "SELECT page_no FROM statement_pages WHERE order_id = ?1 ORDER BY page_no" ).bind(orderId).all<{ page_no: number }>(); const fixed = await plannedLength(orderId, pages.results.length); // computed, not guessed const { readable, writable } = new FixedLengthStream(fixed); const writer = writable.getWriter(); // Do not await this in the handler; it runs while the response is being read. writePdf(writer, orderId, pages.results).catch((err) => { console.error("pdf_stream_failed", { orderId, err: String(err) }); }); return new Response(readable, { headers: { "content-type": "application/pdf", "content-disposition": `attachment; filename="statement-${orderId}.pdf"`, "cache-control": "private, no-store", }, }); }, }; 

Inside writePdf, the invariant to protect is "write, then await the writer's readiness when it is not ready":

async function emit(writer: WritableStreamDefaultWriter<Uint8Array>, bytes: Uint8Array) { // write() returns a promise; awaiting it is what keeps the producer honest. // Ignoring it is how a "streaming" writer still ends up holding everything. await writer.write(bytes); } 

FixedLengthStream deserves its own paragraph because it solves the worst failure mode in this article. Per the Workers documentation, it is an identity stream with a declared length; when used to produce a response it sets Content-Length instead of chunked encoding, and it errors if too many or too few bytes are written through it. That converts "did I write exactly what I promised" from a client-side mystery into a runtime error on your side. The alternative — chunked responses with no length — is perfectly valid, but then truncation is silent unless the format detects it (see the truncation table below).

Images: the decode you cannot see

An image is the clearest example of "output size ≠ live bytes", because the file on disk is compressed and the thing you manipulate is not.

Measured on a 4,000 × 3,000 JPEG, 7,559,511 bytes on disk, 36,000,000 bytes as raw RGB, resized to 1,200 × 900 and saved at quality 82:

Same visible result (0.2% difference in bytes), half the memory, because the JPEG decoder can produce a smaller image directly instead of materialising 12 megapixels first and throwing them away. Two full decodes of that image in one isolate is 163 MB — which is why "image resize" endpoints are such a common source of 1102 in front of a perfectly ordinary phone camera.

Practical rules:

• Never trust the upload. A 4,000 × 3,000 image is 36 MB decoded; a 12,000 × 9,000 one is 324 MB before any library has done anything clever. Set a pixel-count limit (limitInputPixels in sharp-style libraries, an explicit width × height check elsewhere), and reject before decoding.
• Define the size matrix you will serve. On-demand resizing of arbitrary dimensions is an unbounded cache and an unbounded CPU bill; a fixed set (thumbnail, card, detail) is three cache keys.
• Alpha and colour space are memory. RGBA is 4 bytes per pixel; CMYK JPEGs decode differently from RGB; a 16-bit PNG doubles again.
• Decode in the same step that encodes, if the library allows it. A pipeline from file to resized buffer holds one band at a time; a pipeline from "load" to "resize" holds the whole image.

ZIP: the central directory is at the end, so Content-Length is a lie

A ZIP archive is like a PDF in that its index is at the end (the central directory plus the end-of-central-directory record), and unlike a PDF in that the size of each entry is not known until it has been compressed. That combination means:

• You cannot compute Content-Length before producing the archive (compression output size is data-dependent).
• You can stream it, and the format has supported that for decades: set bit 3 of the general-purpose flags, write a zeroed CRC and size in the local header, and append a data descriptor after the data.

Measured, with 1 MB random files (incompressible by design, 1 MB at a time):

The streamed archive is a real archive, not a degenerate one: the general-purpose flag read back as 0x8 (bit 3 set, data descriptor present), testzip() returned None, and all entries listed with correct sizes. The streamed peak did not move between 20 MB and 100 MB of input, which is the property to test for.

What to watch:

• store vs deflate. Deflate on already-compressed media (JPEG, PNG, MP4, PDF) spends CPU and time to make the file slightly larger. Choose per entry type.
• ZIP64. The classic format caps at 4 GB total and 65,535 entries; past that you need ZIP64 records, which change the local headers. If "download all photos" can reach those numbers, decide now, not on the day it happens.
• Filenames are bytes. Non-ASCII names need UTF-8 with bit 11 set, or the recipient's unzip tool mangles them. This is a support ticket generator, not a bug you will see in testing with file-0001.pdf.
• Rate-limit the button. "Download everything" is the single most expensive action a user can take and it arrives in bursts — the person clicks twice, then three colleagues do the same. This is a queue or a cached artifact, not a per-request regeneration.

Backpressure: the producer that ignores the consumer

Streaming fixes the size problem and introduces a pacing problem. A generator that computes faster than the network drains will happily keep producing, and the queue between them grows until it is the file in memory — the same bug, with a different name.

Measured with a producer writing 4,000 chunks of 4,096 bytes (16.0 MB total) against a consumer that takes 0.2 ms per chunk:

440× less memory for 2.2% more wall time. That is the trade, and it is nearly always worth taking. It also explains a class of bug that looks like a memory leak and is not: the generator is fine, the client is slow, and nobody ever told the generator to slow down.

Where the leverage lives in each stack:

• Workers / web streams: await writer.write(chunk) — await it, or read writer.desiredSize and wait on writer.ready before producing the next chunk. TransformStream in Workers is an identity stream (it forwards TypedArray chunks unchanged), so it is a pipe, not a transformer; the pacing contract is still "respect the writable side".
• Node: stream.pipeline() wires drain handling for you; a hand-rolled for await loop that ignores a falsy return from write() does not. If you write to res directly, check res.write()'s return and wait for 'drain'.
• Python: write to a sink that can block, and do not accumulate in a list "just for the checksum". If you are using a generator into a WSGI response, the server's write call is the backpressure signal.

One more consequence worth stating plainly: backpressure makes generation time proportional to download time. The user who walks away mid-download holds your generator open, so you need a timeout and a cleanup path (an AbortSignal, a cancel handler on the readable side, a try/finally around the writer). In Workers, per the docs, when the client disconnects, tasks for that request may be cancelled — plan for your finally block to run at an arbitrary point, and make partial artifacts impossible to mistake for complete ones.

How a stream fails after you have already said 200

This is the design-level difference between buffering and streaming, and it cannot be patched later: once the first byte is out, the status code is spent. With a buffered response you can return 500 with a reason. With a streamed response, a failure at page 900 of 1,200 produces a 200 and a short file.

The runtime gives you one strong tool: FixedLengthStream fails if the byte count does not match, so a truncated body becomes an aborted response rather than a clean end. That is only available when you can compute the total — for a PDF whose layout you control you often can, by taking the page list first and summing the fixed prologue plus the per-object sizes that follow from it; for a ZIP you cannot, because compression output size is data-dependent.

When you cannot declare a length, make the payload self-checking. I truncated four formats at 90% and tried to read them:

CSV is the dangerous one: a table with a missing tail is still a valid table. If your users download CSVs, either compute the length (easy — a CSV you generate is deterministic), or append a final row that carries the record count and expected checksum and make your client's importer check it, or use a container that fails loudly (a zip with one entry costs little and cannot be silently short).

Three more patterns that make mid-stream failure survivable:

• Log the expected and actual byte counts. bytes_written versus content_length is the cheapest detector of this entire class of bug, and it is the one number nobody logs.
• Write a marker last. A one-line manifest or a %%EOF-style terminator that the consumer validates turns a truncated file into a validation error.
• Retry inside the producer, not outside the response. Once headers are sent, an outer retry can only produce a second half-file. Retries belong to the step that fetches data before the first byte is emitted.

Six ways a streaming generator fails

The second row is the one that costs money, because it is invisible to your error rate. Your dashboard shows successes. The customer shows you a PDF that ends mid-table.

What to log, and what to alarm on

For a file endpoint, four numbers per request tell you almost everything:

path, format, output_bytes, bytes_written, duration_ms, peak_hint, pages_or_entries, client_aborted 

• output_bytes vs bytes_written — mismatch means a truncated stream. Alarm on any mismatch.
• peak_hint — you cannot read isolate memory from inside the Worker, but you can log the largest single allocation you made (a page buffer, a decoded image) and the concurrency you allowed. Multiplying those is a usable proxy.
• pages_or_entries vs output_bytes — a change in the bytes-per-page ratio is how you notice that someone's report got 40× bigger.
• client_aborted — count these separately. A high abort rate on big downloads means your users are giving up before the file lands, which is a product problem your error rate will never show.

Alarms worth having: 1102 rate by invocation status (Cloudflare separates exceededMemory from exceededCpu, and the distinction tells you which lever to pull), byte-mismatch count, and p95 time-to-last-byte per format.

Four tests that catch this class of bug

1. The flatness test. Generate the same document at 1×, 4×, and 16× the content and assert that peak memory does not scale. This is the test that separates "streaming" from "streaming-shaped":

import resource, subprocess, sys def peak_mb(pages: int) -> float: out = subprocess.run([sys.executable, "gen.py", str(pages)], capture_output=True, text=True, check=True).stdout return float(out.strip()) small, large = peak_mb(500), peak_mb(8000) # 16x the content assert large - small < 25, f"peak scaled with output: {small:.1f} -> {large:.1f} MB" 

On the streamed writer, 500 pages and 8,000 pages differed by 4.2 MB (9.4 → 13.6 MB, for 16× the content). The same test on the naive writer differs by 37.1 MB (11.4 → 48.5 MB) and fails the assertion, which is exactly what a useful test looks like — it fails on the version you are tempted to write.

2. The write-count test. Wrap the sink, count calls. One write means you are buffering, whatever the API is called.

3. The truncation test. Take a produced file, cut it at 90%, and assert that your consumer detects it. For a format that does not detect it (CSV), assert that your manifest does.

4. The concurrency test. Six simultaneous downloads of the largest supported document, watching peak memory and error rate. If your arithmetic says six will not fit, this is where you find out — before a customer's month-end does.

Eighteen checks before you stream a generated file

1. The maximum output size is written down as a number, with the input that produces it.
2. The fixed cost of the process is measured with an empty request, fonts and tables included.
3. Every font is subset at build time; no font loading happens per request.
4. Every image is decoded at the size you will serve, with a pixel-count limit before decode.
5. The output size is either computable in advance or the format is self-checking.
6. The producer awaits the writer's readiness instead of assuming it consumes instantly.
7. The write count to the sink is greater than one, and you have seen it be greater than one.
8. Peak memory is tested at 1×, 4× and 16× content, and does not scale with output.
9. Concurrency is capped deliberately, and the cap comes from arithmetic, not from a hunch.
10. The largest single object the code allocates is known, named, and bounded.
11. Mid-stream failure is logged with expected versus actual bytes.
12. A truncated file is detectable by the consumer, and you have tested it by truncating one.
13. Client disconnects run a cleanup path, and partial artifacts cannot be mistaken for complete ones.
14. Retries happen before the first byte, never by re-running a response that already started.
15. Large or bursty downloads are queued or cached rather than regenerated per click.
16. Non-ASCII filenames and ZIP flags are tested with a real name, not file-0001.pdf.
17. The output is byte-identical for identical input, so a retry or a range request is safe.
18. Somebody has written down what happens when ten users request last year at 9 a.m. on the first of the month.

Number 18 is the one that has actually saved a system I was responsible for. Everything above it is how you make the answer boring.

What none of this fixes

• A job that needs 5 minutes of CPU does not become a request just because you stream it. Move it to a queue consumer, a container, or a scheduled job that writes to storage.
• A single object larger than the ceiling cannot be produced in RAM no matter how many chunks you write, because some step — decode, parse, render — has to hold it. Chunked generation is the only answer, and some formats do not allow it.
• Memory fragmentation and library overhead are real; your arithmetic will be optimistic by 10–30%, which is why the number to design against is not 128 MB but something like 96.
• The client's own memory. A 40 MB PDF opened in a phone browser is a different outage, and the size limit you choose should account for who downloads it.
• A streaming response that computes slowly is still bounded by CPU time, and a user who abandons it is a user who saw nothing.

The useful mental shift is this: a generated file is not a value your function returns, it is a process that occupies memory over time, shared with everyone else on the machine, in a runtime that will not tell you it is full until it is. Decide how big it can get, make the live set independent of that size, and put a byte count on both ends. The file format is the easy part.

Disclosure: I build small automation tools and write about the parts that break in production, including this one. I also keep an English guide for an AI assistant app whose invite system gives me a concrete example of a service that must serve small pages to many people at once: Pickle invite code — what the app does and how to enter it and the invite link pickle.com/i/RDKXK77G9ZRT. If someone joins through that link I receive in-app usage credit; it costs the person joining nothing. No part of this article depends on that product, and every pattern above is generic.