I recently built a bulk file downloader: a list of URLs in, every file stored, plus one ZIP of everything at the end. The ZIP part turned out to need no library at all on modern Node, and the downloading part had more traps than I expected. Here are both. Disclosure: the finished tool is linked at the end, and I built it.

A ZIP writer in about 60 lines

A ZIP file is simple: for each file, a 30-byte local header, the name, and the data. Then a central directory with a 46-byte entry per file, and a 22-byte end record pointing at it. Every number is little-endian.

The only hard part used to be CRC-32, and Node 22.2 added zlib.crc32. With deflateRawSync for compression, that's everything:

import { crc32, deflateRawSync } from 'node:zlib'; export function buildZip(entries, now = new Date()) { const time = (now.getUTCHours() << 11) | (now.getUTCMinutes() << 5) | (now.getUTCSeconds() >> 1); const date = ((now.getUTCFullYear() - 1980) << 9) | ((now.getUTCMonth() + 1) << 5) | now.getUTCDate(); const locals = []; const centrals = []; let offset = 0; for (const { name, data } of entries) { const nameBuf = Buffer.from(name, 'utf8'); const deflated = deflateRawSync(data); const useDeflate = deflated.length < data.length; // JPGs and PNGs are already compressed const body = useDeflate ? deflated : data; const crc = crc32(data) >>> 0; const head = Buffer.alloc(30); head.writeUInt32LE(0x04034b50, 0); // local file header signature head.writeUInt16LE(20, 4); // version needed head.writeUInt16LE(0x0800, 6); // flag: names are UTF-8 head.writeUInt16LE(useDeflate ? 8 : 0, 8); head.writeUInt16LE(time, 10); head.writeUInt16LE(date, 12); head.writeUInt32LE(crc, 14); head.writeUInt32LE(body.length, 18); // compressed size head.writeUInt32LE(data.length, 22); // uncompressed size head.writeUInt16LE(nameBuf.length, 26); locals.push(head, nameBuf, body); const cen = Buffer.alloc(46); cen.writeUInt32LE(0x02014b50, 0); // central directory signature cen.writeUInt16LE(20, 4); cen.writeUInt16LE(20, 6); cen.writeUInt16LE(0x0800, 8); cen.writeUInt16LE(useDeflate ? 8 : 0, 10); cen.writeUInt16LE(time, 12); cen.writeUInt16LE(date, 14); cen.writeUInt32LE(crc, 16); cen.writeUInt32LE(body.length, 20); cen.writeUInt32LE(data.length, 24); cen.writeUInt16LE(nameBuf.length, 28); cen.writeUInt32LE(offset, 42); // where this file's local header starts centrals.push(cen, nameBuf); offset += head.length + nameBuf.length + body.length; } const cd = Buffer.concat(centrals); const end = Buffer.alloc(22); end.writeUInt32LE(0x06054b50, 0); end.writeUInt16LE(entries.length, 8); end.writeUInt16LE(entries.length, 10); end.writeUInt32LE(cd.length, 12); end.writeUInt32LE(offset, 16); return Buffer.concat([...locals, cd, end]); }

Limits: this is classic ZIP, not ZIP64, so stay under 4 GB and 65,535 files, and it builds in memory. I capped mine at 200 MB. I tested it by extracting the output with Windows' built-in Expand-Archive, which rejects bad CRCs, and comparing every file's SHA-256 with the original.

Four bugs from downloading files in bulk

1. Duplicate detection raced itself. I hashed each file with SHA-256 and skipped hashes I'd already stored. With ten downloads in parallel, two copies of the same PDF both checked the map before either finished uploading, so both were stored. The fix is to claim the hash (and the storage key) synchronously, before the first await:

const key = storageKey(fileName, contentType, taken); hashes[sha256] ??= key; // claim first await store.setValue(key, data, { contentType }); // then upload

2. Don't trust Content-Type. Plenty of servers send images as application/octet-stream or binary/octet-stream. Check the first bytes instead: ffd8ff is JPEG, 89504e47 is PNG, %PDF is PDF, RIFF....WEBP is WebP. Then use the URL's extension, and only then the header.

3. Cap the size while streaming, not after. Content-Length is often missing, so read res.body chunk by chunk and cancel once you pass the limit. Otherwise one 4 GB video takes down the whole run.

4. Your storage may "helpfully" parse files. When building the ZIP, I read each stored file back from Apify's key-value store, and the JSON file came back as a parsed object rather than bytes, which crashed deflateRawSync. Ask for the raw buffer explicitly (getRecord(key, { buffer: true }) in the Apify client).

The finished tool

All of this is packaged as Bulk File & Image Downloader on Apify: paste URLs, or point it at another scraper's dataset, and you get every file stored with its name, type, size and SHA-256, duplicates skipped, optional type and size filters, retries on 429 and 5xx errors, and ALL_FILES.zip at the end. It costs $2 per 1,000 files, and failed, skipped and duplicate URLs are free.