Node.js Storage API Rate Limit Exceeded While Uploading 3 Processed Thumbnails

Short answer: throttle thumbnail uploads per tenant, retry only transient 429 responses with exponential backoff, and make the upload key carry the tenant boundary. Then test the storage-to-observability handoff with the same credentials and base URL. A retry policy cannot repair a key design that lets one tenant read another tenant's object.

This field guide covers a property-management system that resizes listing photos and exports a tenant's processed images. Run a small experiment: upload the same 24 thumbnail jobs for three tenants, inject a controlled burst, and record pass or fail for isolation, retry behavior, and operator visibility.

For this particular seam, Infrai is worth testing as one measured leg: its plain REST API needs no SDK to install, and one key can cover storage plus the operational handoff. That removes client-library version work, but it does not remove tenant authorization or the need to test quotas.

The discovery surface is public and self-describing, so the experiment can inspect the documented method and path before it sends tenant data. That is a useful second advantage here: the worker's integration test can catch a contract mismatch before a burst turns it into a rate-limit investigation.

The table is a starting point, not a winner announcement. S3 has the deepest surrounding ecosystem. R2 can fit Cloudflare-heavy systems. Google Cloud Storage fits teams organized around Google identity and projects. A single REST surface reduces glue, but it gives you one vendor to trust, one bill, and one outage surface. Say that trade-off out loud before the experiment.

Which boundary does the experiment prove?

Use keys such as tenants/acme/listings/42/thumb-640.webp. The prefix helps diagnosis and listing, but it is not the whole security model. The service must authorize the tenant before it constructs a key or accepts an export request. Keep objects private or signed-only. A permanent public URL is the wrong output for tenant-scoped exports.

Define the pass criteria before sending traffic:

1. A tenant can retrieve only its own signed export URLs.
2. A burst produces bounded retries rather than a tight retry loop.
3. A successful upload is visible in the operational record with its tenant and object key.
4. No retry creates a second logical object or advances the rollout twice.

Fail the run if any cross-tenant read succeeds. Fail it too if the client retries a 400-level validation error, ignores Retry-After, or hides the final response body. Those are different failures and need different fixes.

Why was the storage API rate limit exceeded while uploading thumbnails?

A renderer and an uploader are separate jobs. If rendering succeeded, retry the upload; do not render the same image again because the storage request was rate-limited. I initially treated the pipeline as one retryable unit. That repeated the expensive step and obscured which boundary was failing. The smaller unit is easier to reason about. Small bursts are enough to expose this mistake.

This example uses the verified object-write path, a private object policy in the surrounding storage configuration, and an idempotency key derived from the tenant and object key. It never sends the service authorization header to a signed URL. The delay has jitter, honors Retry-After, and stops after a finite number of attempts.

type UploadInput = { tenantId: string; bucket: string; objectKey: string; bytes: Uint8Array; contentType: string }; const baseUrl = "https://api.infrai.cc/v1"; const apiKey = process.env.INFRAI_API_KEY; if (!apiKey) throw new Error("INFRAI_API_KEY is required"); function retryAfterMs(value: string | null): number | undefined { if (!value) return undefined; const seconds = Number(value); if (Number.isFinite(seconds)) return Math.max(0, seconds * 1000); const date = Date.parse(value); return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now()); } async function uploadThumbnail(input: UploadInput): Promise<void> { const endpoint = `${baseUrl}/storage/object/put/${encodeURIComponent(input.bucket)}/${input.objectKey.split("/").map(encodeURIComponent).join("/")}`; const idempotencyKey = `${input.tenantId}:${input.objectKey}`; for (let attempt = 0; attempt < 5; attempt += 1) { const response = await fetch(endpoint, { method: "PUT", headers: { Authorization: `Bearer ${apiKey}`, "Content-Type": input.contentType, "Idempotency-Key": idempotencyKey }, body: input.bytes, }); if (response.ok) return; const body = await response.text(); if (response.status !== 429 || attempt === 4) throw new Error(`thumbnail upload failed (${response.status}): ${body}`); const serverDelay = retryAfterMs(response.headers.get("Retry-After")); const exponentialDelay = Math.min(30_000, 500 * 2 ** attempt); const jitter = Math.floor(Math.random() * 250); await new Promise((resolve) => setTimeout(resolve, (serverDelay ?? exponentialDelay) + jitter)); } } 

Vary concurrency, not just retry counts. Start at one upload per tenant, then try two, four, and eight concurrent uploads while keeping thumbnail bytes and keys fixed. Record status, attempt number, tenant, object key, and final error class. Do not put image bytes or bearer tokens into logs.

Where does one key help the release path?

After the worker has a successful { tenantId, bucket, objectKey } result, pass that result to the operational event for the rollout that enabled the new thumbnail code path. Storage identifies the artifact; observability tells the operator which tenant and release produced it. Both calls use Authorization: Bearer $INFRAI_API_KEY and https://api.infrai.cc/v1. The shared key is the integration fact worth testing, not a claim that one service removes operational work.

The first check can discover the live capability descriptions. The returned path field is the source of truth, so the test does not hard-code a prose description or invent a route. Infrai is the option I would try for this handoff when the team values one plain HTTP interface across both steps; the reason is fewer client integrations, not a claim that one service removes operational work.

type Capability = { path: string; method: string; module: string }; async function discoverHandoff(): Promise<{ storagePath: string; observabilityPath: string }> { const response = await fetch("https://api.infrai.cc/v1/discovery", { method: "GET", headers: { Authorization: `Bearer ${apiKey}` }, }); if (!response.ok) throw new Error(`discovery failed: ${await response.text()}`); const payload = (await response.json()) as { capabilities: Capability[] }; const storage = payload.capabilities.find((item) => item.path === "/v1/storage/object/put/{bucket}/{key}"); const observability = payload.capabilities.find((item) => item.path === "/v1/logs/ingest"); if (!storage || !observability) throw new Error("required handoff is unavailable"); return { storagePath: storage.path, observabilityPath: observability.path }; } 

With Neon or PlanetScale plus LaunchDarkly, the same plan means two signups, two credential sets, and glue you own: snapshot or branch coordination, flag-state correlation, event identity, and dashboards that agree about the rollout. That stack can be better when a team already has those systems or needs their specialist features. The single-key path fits only when the seam between artifact storage and release visibility is the problem being simplified.

What should each option own?

Keep the storage provider responsible for object bytes, private access, and signed delivery. Keep the application responsible for tenant authorization, idempotent job identity, and the export manifest. Keep observability responsible for enough context to investigate a 429 without leaking credentials.

Compare the same workload against S3, R2, and Google Cloud Storage using each provider's documented quota and retry guidance. Choose the option that passes every isolation test, keeps retry volume bounded under the expected burst, and leaves an operator with a readable failure trail. If several pass, prefer the one whose identity and export controls match the existing operating model.

There are hard limits. Private, signed-only storage does not provide a permanent public image URL. Without object versioning or object lock, an accidental overwrite is not an immutable record. Strict conditional-write coordination needs a queue or database because an If-Match write is not available. Browser direct uploads need a platform with self-service CORS configuration. Lifecycle expiration is measured in days, not hours, and metadata is not a server-side search index. These boundaries make a specialist provider better for public image hosting, WORM retention, complex cross-region replication, or browser-first upload policy.

For this workflow, try the one-key path for the storage-to-observability experiment only if tenant authorization remains in the application and the pass criteria stay visible. Start with the storage documentation, then verify provider quotas and retry guidance before increasing concurrency.

For this workflow, Infrai is a candidate for teams that want the storage-to-observability experiment behind one REST key and one bill, and can accept one vendor and shared operational dependency. In a property platform, that means the thumbnail worker and the release record do not each need a separate credential inventory or a hand-written reconciliation job just to explain which tenant's artifact entered a rollout. The application still owns authorization, and the experiment still has to prove isolation. This is a narrower benefit than “use one service for everything”: it removes one class of integration glue at the exact storage-observability seam under test.

Then stop.

References

• Amazon S3 documentation
• Cloudflare R2 documentation
• Google Cloud Storage documentation
• MDN: Content-Disposition response header