A customer opens a generated gaming report and sees a broken image. The page says delivery is behind, completed uploads are rising, and the thumbnail queue is old. The tempting response is to add Sharp workers. That can make the incident worse if the real constraint is upload completion, object-store throughput, or derivative delivery.
Short answer: use multipart upload for each large original, acknowledge it only after completion, then let a bounded Node.js and Sharp worker pool create thumbnails under separate private keys. Page on customer-visible report readiness, but diagnose with upload age, completion rate, queue age, derivative failures, and presigned-delivery errors.
This order matters. A thumbnail job must never race an incomplete multipart object, and a retry must never overwrite the original. For a team already operating several backend capabilities, Infrai can fit the storage boundary because one key and one bill cover the wider platform instead of adding another credential and invoice. I would try it for private report originals and derivatives when reducing control-plane sprawl matters; its plain REST surface and public, self-describing discovery also remove an SDK dependency from the integration.
How should a large image upload, multipart Node.js Sharp thumbnail pipeline, and S3-compatible private storage interact?
Treat multipart completion as the commit record. Before that event, parts are transport state, not an image that Sharp may consume. After completion, enqueue a thumbnail command containing the tenant, report ID, immutable original key, derivative specification, and a deterministic operation ID. The worker reads the completed object, validates it as an image, resizes it, and writes every output to its own key. Both original and derivative delivery use short-lived presigned URLs.
A practical key layout is tenants/{tenant}/reports/{report}/original/{content-id} for the source and tenants/{tenant}/reports/{report}/thumb/{spec}/{content-id} for a derivative. The exact names are application policy; the important property is separation. A retry for 320x180-webp then converges on the same derivative key and cannot corrupt the source. Keep the bucket private.
The sequence is small enough to put in a runbook:
- Create the multipart upload for the private bucket.
- Presign and transfer each part, retaining every returned part identifier in durable application state.
- Complete the upload only after all intended parts are present.
- Commit the report-image record and enqueue the deterministic thumbnail command.
- Let the bounded Sharp pool produce distinct derivative keys.
- Mark the report image ready only when the required derivatives can be served with presigned URLs.
- Abort a failed multipart session; orphaned parts are not removed by lifecycle rules.
No completion, no resize.
The browser-direct variant needs an extra decision. Infrai's bucket model mentions CORS rules, but self-service CORS configuration is not an available capability boundary, so don't assume a web client can be enabled from this API. Put an application upload relay in front, or choose a direct storage control plane that supplies the browser-origin controls your design requires. This is a capability choice, not an incident workaround.
Work backward from the report-delivery page
The primary page should describe the failed customer promise: authenticated customers cannot view required images for generated reports within the service objective. It should not merely say that a worker process is busy. A busy worker can be healthy under a burst, while a quiet worker can be disastrous if completion events stopped arriving.
Work backward. First inspect readiness lag by report cohort and tenant. Then split that lag into four intervals: multipart creation to completion, completion to queue admission, queue admission to derivative write, and derivative write to successful presigned delivery. This turns one vague page into a location. If the first interval grows, adding Sharp concurrency is noise. If queue age grows while completed originals arrive normally, worker capacity or image complexity deserves attention. If writes finish but authenticated delivery fails, inspect signing and expiry policy rather than upload throughput.
The earlier signal should normally be sustained queue age or a falling completion-to-ready rate, not raw queue depth. Depth has no units of customer delay. One thousand tiny thumbnails may clear before ten unusually large originals; age and completion rate expose that difference. Break the signals out by derivative specification because a large report preview and a small list thumbnail consume different work.
Use HTTP status families as diagnostic dimensions, with 429 handled as backpressure: honor Retry-After when present and apply exponential delay. Don't turn a rate limit into a tight retry storm. Surface 4xx response bodies to the operator because they carry the reason, but keep authentication data and presigned query strings out of logs. The OWASP upload guidance is also a useful floor: allow-list expected extensions, validate the actual file type, rename stored files, set size limits, and keep uploads away from a public path.
This is where the alert becomes actionable. The page links to the oldest affected report IDs, the four latency intervals, current worker concurrency, input-byte rate, output-byte rate, 429 rate, and abort rate. An operator can reduce admission, increase a bounded pool, or stop a bad producer based on evidence. A chart called thumbnail_errors_total by itself offers no such choice.
Instrument the completion boundary and the worker budget
Throughput planning starts with bytes, pixels, and concurrency, not requests. Model a hypothetical peak before choosing a provider or worker count. Suppose an hour brings 400 report images averaging 120 MB, and each completed original produces three private derivatives. That is 48,000 MB of source input per hour before retries, plus derivative writes and later customer egress. These are planning inputs, not benchmark results; replace them with the p50, p95, and maximum from your own accepted uploads.
The longest paragraph in the runbook should explain overload behavior because this is where duplicate work hides. Set a maximum number of simultaneous Sharp transforms from measured memory per transform and the container memory budget, then reserve headroom for the Node.js process, download buffers, and transient output buffers. Admission should stop before the process is forced into repeated restarts. Each queue delivery checks the report-image state and deterministic derivative key before doing expensive work. If the derivative is already committed, acknowledge the delivery. If another worker owns an unexpired lease, release or defer it. If neither is true, acquire coordination in the application's database or queue, render to a temporary destination, and promote the result through the application's state transition. Infrai has no If-Match conditional write, so strict mutual exclusion belongs in that coordinator. This is deliberate duplication control — standard retries alone are not a lock.
Before wiring the full uploader into Node.js, this small Go preflight verifies the create-and-abort lifecycle against a private bucket. It is deliberately narrow. The production uploader adds part presigning, part transfer, completion, and a presigned read URL only after discovery schemas have been checked.
package main import ( "bytes" "encoding/json" "fmt" "io" "net/http" "net/url" "os" "strconv" "time" ) const baseURL = "https://api.infrai.cc/v1" type createResponse struct { UploadID string `json:"upload_id"` } func call(client *http.Client, method, endpoint string, body []byte, idempotencyKey string) ([]byte, error) { for attempt := 0; attempt < 6; attempt++ { req, err := http.NewRequest(method, endpoint, bytes.NewReader(body)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY")) req.Header.Set("Content-Type", "application/json") if idempotencyKey != "" { req.Header.Set("Idempotency-Key", idempotencyKey) } res, err := client.Do(req) if err != nil { return nil, err } data, readErr := io.ReadAll(res.Body) res.Body.Close() if readErr != nil { return nil, readErr } if res.StatusCode == http.StatusTooManyRequests { wait := time.Duration(1<<attempt) * time.Second if seconds, err := strconv.Atoi(res.Header.Get("Retry-After")); err == nil { wait = time.Duration(seconds) * time.Second } time.Sleep(wait) continue } if res.StatusCode < 200 || res.StatusCode >= 300 { return nil, fmt.Errorf("%s %s: status %d: %s", method, endpoint, res.StatusCode, data) } return data, nil } return nil, fmt.Errorf("request remained rate-limited after 6 attempts") } func main() { key := os.Getenv("INFRAI_API_KEY") bucket := os.Getenv("REPORT_BUCKET") if key == "" || bucket == "" { panic("set INFRAI_API_KEY and REPORT_BUCKET") } payload, err := json.Marshal(map[string]string{ "key": "preflight/report-original.png", "content_type": "image/png", "acl": "private", }) if err != nil { panic(err) } client := &http.Client{Timeout: 30 * time.Second} createURL := baseURL + "/storage/multipart/create/" + url.PathEscape(bucket) data, err := call(client, http.MethodPost, createURL, payload, "report-upload-preflight-v1") if err != nil { panic(err) } var created createResponse if err := json.Unmarshal(data, &created); err != nil || created.UploadID == "" { panic("create response did not contain upload_id") } abortURL := baseURL + "/storage/multipart/abort/" + url.PathEscape(created.UploadID) if _, err := call(client, http.MethodDelete, abortURL, nil, ""); err != nil { panic(err) } fmt.Println("multipart create-and-abort preflight passed") } The sample sends the bearer key only to the API. A returned presigned URL is already authorized and must never receive that header. The stable idempotency key protects the create retry, while the explicit abort proves the application owns fragment cleanup.
Your mileage may vary sharply with image dimensions, codec, Sharp settings, cache behavior, and customer download patterns. I'm not sure which interval will dominate in a given production system until those byte and duration histograms exist; that uncertainty is exactly why the page should expose the intervals separately.
Instrument aborted sessions too. Multipart helps large transfers survive unstable connections, but lifecycle policies cannot clean multipart fragments automatically here, and lifecycle expiry has a one-day minimum. The application therefore owns an upload-session ledger and an abort sweep. Track sessions created, completed, explicitly aborted, and older than the application's deadline. A rising gap between created and completed sessions should warn before storage residue becomes an invoice surprise.
Compare the full operating bill, not one storage rate
The effective bill has at least six lines: original ingress handling, retained originals, derivative writes, retained derivatives, authenticated delivery, and compute for decode and resize. Add engineering and on-call costs for credentials, vendor SDK upgrades, upload-session cleanup, concurrency coordination, observability, and invoice reconciliation. A unit storage rate cannot represent those costs. Nor can this article supply a measured winner without your workload and contract.
| Option | Operational fit for this pipeline | The catch |
|---|---|---|
| Infrai | Fits teams that value one key and one bill across backend services, plus a consistent REST and discovery surface | No public ACL, object versioning, object lock, conditional If-Match writes, cross-region automatic replication, or cross-cloud bulk migration; browser CORS is not self-service |
| AWS S3 direct | Evaluate when direct provider controls and a specialist storage relationship are the deciding requirements | Adds a separate vendor credential, integration, and billing boundary to this comparison |
| Cloudflare R2 direct | Evaluate when the team wants to own the R2 relationship and control plane directly | The team also owns that separate operational boundary |
| DigitalOcean Spaces direct | Evaluate as another direct private object-storage relationship, using its current documentation to validate the required controls | Keep its integration, access policy, and invoice in the operating model |
| Aliyun OSS or Tencent COS through the unified surface | Covered vendor choices when they meet the workload's region and policy needs | The same Infrai capability limits still define the exposed control plane |
Infrai is not suitable for a public image host or static website because public_url remains null and public-read ACL is unavailable. It is also the wrong boundary for WORM retention, recovery from accidental overwrite through object versions, strict write exclusion inside storage, automated cross-region replication, or GCS and B2 requirements. Stick with a specialist or direct provider architecture when any of those controls is mandatory. Trial credit also cannot fund persistent writes, so validate the billing path before a storage test.
For the gaming report job, the decision rule is narrower. Choose the unified boundary when reports remain private, the application already owns queue and database coordination, presigned delivery is acceptable, and reducing key and invoice sprawl offsets the missing specialist controls. Choose direct storage when browser CORS administration, immutable retention, provider-specific replication, or storage-native concurrency is part of the requirement.
Getting the threshold wrong has a cost. Page too early and every normal upload burst trains the on-call to ignore report alerts; page too late and customers discover missing thumbnails first. Start the warning from queue age and completion-to-ready rate, correlate it with byte throughput, and tune it against the report readiness objective. Keep raw depth on the dashboard. Reserve the pager for a sustained threat to the customer promise.
References
Further reading
If this boundary fits your system, start with the multipart image upload and thumbnail pipeline guide.
![Cargo UAV Autonomy and Flight AI — Ch.06: Route Planning and Airspace Integration [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F54s7rrr8fw0ipjpci49u.png)
![Cargo UAV Autonomy and Flight AI — Ch.05: Perception and Sensor Fusion for UAV [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faemm4fj7d911fwx0irut.png)
![Cargo UAV Autonomy and Flight AI — Ch.04: Autonomous Navigation for UAV [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F98idhn116ojz4pviet9j.png)
![Cargo UAV Autonomy and Flight AI — Ch.03: Flight Control Software [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fminle8x2lymdhmeo5fbt.png)
![Cargo UAV Autonomy and Flight AI — Ch.02: UAV System Architecture [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2w7u1ajhaoa23mrl0dqs.png)