A 1 GB archive that restarts at 99% is an operational problem, not a progress-bar problem. Short answer: use multipart upload when a whole-file retry exceeds the recovery budget or the client must resume; use a single transfer only when a verified restart is acceptably cheap. For private document storage, make the decision before the first byte moves, then prove the stored object exists before OCR, parsing, or embeddings begin.
The tempting implementation is one upload endpoint and a size cutoff copied from a blog post. It looks fine in a notebook with a clean connection. It fails under a browser reload, a laptop sleep, an expired authorization, or a duplicate completion request. For an AI application, the damage can continue after storage: an unverified retry can trigger another parse and another round of token-producing work.
This is an experiment note, so the result has a constraint: the transfer mode must pass a recovery evaluation for the actual client population. There is no universal multipart threshold for a 100 MB PDF or a 1 GB ZIP. A file's retry cost, the connection behavior, and the persistence available to the client matter more than a round number.
Recovery is the test.
How should Node.js handle PDF and ZIP uploads to private object storage?
Start by separating authorization, byte transfer, verification, and release. A Node.js service can issue short-lived permission scoped to one object key and operation. The client transfers bytes directly to a private object store, records progress locally, and reports a completion attempt. A control-plane worker then reads the object through an independent authenticated path, compares its observed size with the expected manifest, and emits the document-ready event only after that check.
That last transition is where a simple design gets fragile. A successful completion response says a request was accepted; it does not make a browser callback proof that a downstream reader can retrieve the intended object. The ready event needs a stable identity so a repeated completion attempt cannot create a second parser run. Keep the object private by default, test that tenant boundaries reject cross-tenant reads with an expected 403, and avoid placing access credentials in logs.
The size labels are useful planning inputs, not policy. A 100 MB PDF may be cheaper to resend than to coordinate in parts on a stable office network. A 1 GB ZIP will often justify resumability because retrying it consumes more time and connection opportunity, yet a client that cannot persist an upload session has no reliable way to resume it. Measure first.
| Transfer path | Choose it when | State you must own | Evaluation interruption |
|---|---|---|---|
| One transfer | A full retry fits the recovery budget | Object key and expected size | Disconnect close to completion |
| Multipart | Restarting all bytes is too costly or resumption is required | Upload session, completed parts, expiry, and reconciliation | Restart the client after several parts |
| Application proxy | Bytes require inline policy inspection | Stream capacity and backpressure | Keep uploads running while workers are saturated |
The catch is that multipart is not suitable when nobody owns expiry and cleanup of abandoned sessions, or when clients cannot keep the state needed to continue. Keep a one-transfer path for small, repeatable uploads that meet the measured budget. Use an application-mediated stream when policy requires inspection before final storage, while accepting and load-testing the capacity it consumes.
The experiment should measure verified readiness, not upload success
The failed/simple approach is to mark a document ready as soon as the client reports completion. The chosen approach treats readiness as a separate, idempotent state transition. This is less flashy, but it gives a model pipeline a dependable input contract: one logical document version becomes one verified object and one release event.
A focused harness can simulate interruption at three points: before bytes arrive, after a few parts are durable, and immediately after a completion attempt. For every run, assert that the verification reader sees the expected key and byte count; assert that replaying completion produces no second release; and assert that an expired authorization cannot create another object. ZIP handling belongs after that verification. Upload validity does not prove an archive is safe to expand, so impose separate limits for expansion, entry count, and the parsers allowed to inspect content.
For a useful failure-injection run, create a manifest with a known expected byte count and a new document version, authorize exactly that key, and begin the transfer. Interrupt it once before any durable byte, once after the coordinator has recorded several completed parts, and once after the client sends its completion attempt but before it observes a response. Restart the client from its persisted state each time. The acceptance criteria are intentionally about effects: a completed run has one object visible to the independent reader at the expected size; a resumed run does not resend already acknowledged parts; a replayed completion yields the same logical ready-event identity; and an abandoned session reaches the lifecycle path the team has assigned. Run the same cases for a 100 MB PDF and a 1 GB ZIP, but do not infer a universal threshold from those two fixtures. The report should preserve the transfer mode, client network class, restart point, observed bytes resent, and time to verified readiness, because those are the inputs that make a later selection review possible.
No shortcut here.
The following Python model keeps the storage API deliberately generic. It is meant to make the state boundary testable, not to prescribe an SDK or an endpoint shape.
from dataclasses import dataclass from enum import Enum from typing import Protocol class TransferMode(str, Enum): SINGLE = "single" MULTIPART = "multipart" @dataclass(frozen=True) class UploadPlan: key: str expected_bytes: int mode: TransferMode @dataclass(frozen=True) class StoredObject: key: str size_bytes: int class ObjectReader(Protocol): def inspect(self, key: str) -> StoredObject | None: ... class EventWriter(Protocol): def publish_once(self, event_id: str, key: str) -> None: ... def choose_transfer( expected_bytes: int, retry_budget_bytes: int, needs_resume: bool, ) -> TransferMode: if needs_resume or expected_bytes > retry_budget_bytes: return TransferMode.MULTIPART return TransferMode.SINGLE def verify_then_release( plan: UploadPlan, reader: ObjectReader, events: EventWriter, ) -> StoredObject: stored = reader.inspect(plan.key) if stored is None: raise ValueError("verified object was not found") if stored.size_bytes != plan.expected_bytes: raise ValueError("stored size differs from the upload manifest") event_id = f"document-ready:{plan.key}:{plan.expected_bytes}" events.publish_once(event_id, plan.key) return stored One small choice matters here: retry_budget_bytes is evaluated configuration, not a magic constant embedded in clients. It can be derived from the maximum acceptable restart time and observed throughput, then versioned beside the test corpus. I'm not sure a single threshold will remain useful across mobile and managed-office clients; collecting connection and restart outcomes is what resolves that uncertainty.
What should an AI document pipeline measure before choosing multipart upload?
Track time from authorization to verified readiness, total bytes sent per accepted object, resume success after a client restart, expired sessions, and duplicate ready events. Report them by file type and size band, including the PDFs and ZIPs that cluster near your planned boundary. The experiment is incomplete if it reports only successful uploads. Failed and interrupted paths are the part that decides recovery behavior.
Then extend the evaluation into the model pipeline. Count logical parses per verified object version, OCR starts, chunking runs, embedding input, and prompt input created by retries. This is the notebook-to-production connection: a transfer policy should predict downstream work, not merely make an upload chart look tidy. A duplicate event may be cheap in storage and still expensive in a retrieval pipeline.
Use fixtures that are approved for testing and represent the formats your service accepts. Exercise private reads as both the owning tenant and a different tenant. Test authorization expiration, client restart, repeated completion, and reconciliation of unfinished multipart sessions. For systems operating under public-sector requirements, map the selected service and operating controls to the relevant authorization program rather than interpreting a private bucket setting as a compliance result.
The architecture choice is a recovery contract
The recommended transfer mode is the simplest one that passes the recovery and verification evaluation. Don't promote a 100 MB or 1 GB boundary from a first experiment into a permanent rule. Re-run it when client networks, archive sizes, retention policy, or the downstream parsing stack changes.
Private object storage protects access only when authorization scope, tenant isolation, retention, and audit needs are designed alongside it. Multipart protects work only when its session state can be resumed and cleaned up. Those are different contracts, and treating them separately makes document ingestion much easier to reason about.


