Short answer: send originals directly from the client to private object storage, record immutable object identities in application data, generate thumbnails and other variants asynchronously, and issue short-lived signed links only after authorization. Apply lifecycle cleanup by object class and state, not by a single age rule. This keeps large media bytes off the app servers while preserving a recovery path for interrupted work.

The bill has four parts: bytes retained over time, write and read operations, bytes delivered, and the compute used to create derivatives. For an image generation developer tool, the dominant term depends on behavior. A private internal gallery may be driven by retained originals; a public preview feed may be driven by delivery. Measure both before choosing a layout.

What storage pattern should an AI image generation app use for private originals and variants?

Use one private object namespace with separate, stable prefixes for originals and derived assets, plus a relational asset record that owns status and authorization. A bucket is a durability boundary, not a user database. Tenant access belongs in the application layer, and every signed link should be minted only after that layer confirms the caller can read the asset.

A practical object layout looks like tenant/{tenant_id}/asset/{asset_id}/original/{revision} for the source and tenant/{tenant_id}/asset/{asset_id}/variant/{spec_hash} for each derivative. The asset_id is generated before upload. The variant key uses a hash of transformation inputs such as width, output format, and encoder policy, so retrying the same job targets the same object instead of producing a second copy. Don't put user filenames, email addresses, prompts, or other mutable metadata in keys.

The upload path is deliberately split. First, the client asks the application for upload authorization and receives a narrowly scoped signed write. Second, the client sends bytes straight to object storage. Third, the client reports completion, and a worker verifies object metadata before changing the asset from pending_upload to ready_for_derivation. Multipart upload belongs in this data path for large media because individual parts can be retried without routing the whole payload through an application process. The exact multipart limits and completion semantics are provider-specific, so confirm them in the storage service documentation rather than baking assumptions into a shared client.

Keep the app out of the byte stream.

The state transition matters as much as the signed write. A completion callback is a claim, not proof that the expected object exists. The worker should inspect the stored object's size and checksum metadata available from the chosen service, compare them with the upload session, and then enqueue variant work. I treat a missing verification step like an OTP marked delivered merely because an API accepted it: the acceptance event and the completed outcome are different states.

Price the bytes before choosing retention

Start with a workload model rather than a provider price page. Let O be average original bytes, V the sum of retained variant bytes, N the number of assets created each month, and R the average retention in months. A steady-state capacity estimate is N x (O + V) x R, before accounting for versioning, abandoned multipart parts, replicas, or deleted-object retention. Delivery is a separate estimate: views multiplied by the average representation served. Consider a planning example, not a benchmark: 100,000 assets per month, a 12 MB original, and three retained variants totaling 2.1 MB. One cohort occupies 1.41 TB in decimal units. At six months of retention, six equal cohorts occupy about 8.46 TB before overhead. If the thumbnail is 0.1 MB and receives 20 views while the original is rarely fetched, thumbnail delivery is 200 TB per monthly cohort of 100,000 assets? No. That tempting calculation is off by three decimal places; it is 200 GB. Unit mistakes like that can dominate a storage review, so keep units in the model and make another engineer reproduce the arithmetic. The change that usually moves retained capacity is not compressing a tiny thumbnail harder. It is deciding whether every original and every superseded variant deserves the same lifetime. Keep the current gallery derivative while the asset is active, expire replaceable intermediate outputs sooner, and retain originals only for the period justified by regeneration needs and user expectations. Lifecycle management can transition or expire objects according to rules, but application state still has to explain why an object exists and whether deletion is allowed. AWS documents lifecycle actions and rule behavior; Google Cloud Storage documentation covers its storage and lifecycle facilities. Those are implementation references, not a substitute for the product's retention policy.

There is a real catch. Shorter original retention lowers stored bytes, but after deletion you can't create a new crop, encoder format, or higher-resolution derivative from that source. Regeneration may also fail to reproduce the same image if the model, parameters, or dependencies change. If exact future reprocessing is a product requirement, keep the original and accept its storage and governance cost. If the product promises only delivery of current outputs, deleting the source after a stated recovery window can be the cleaner contract. I'm not sure which side is right without the app's deletion promise and re-edit rate; those two measurements settle the decision.

Make variant generation idempotent

A derivative worker should consume an asset identity and a transformation specification, not an arbitrary source URL. It derives the deterministic destination key, checks current database state, reads the private original with service credentials, writes the variant, and commits the resulting metadata. A retry repeats that operation against the same key.

from dataclasses import dataclass from hashlib import sha256 import json @dataclass(frozen=True) class VariantSpec: width: int output_format: str encoder_policy: str def variant_key(tenant_id: str, asset_id: str, spec: VariantSpec) -> str: canonical = json.dumps(spec.__dict__, sort_keys=True, separators=(",", ":")) spec_hash = sha256(canonical.encode("utf-8")).hexdigest()[:20] return f"tenant/{tenant_id}/asset/{asset_id}/variant/{spec_hash}" 

The database row should distinguish pending_upload, deriving, ready, delete_requested, and deleted. Use a uniqueness constraint on asset plus specification hash. Claim jobs with an atomic state change, and make the final database update conditional on the expected prior state. This prevents a slow retry from reviving an asset after a deletion request.

Deletion deserves the same care as creation. Mark the asset delete_requested first, stop issuing new read links, cancel or fence pending derivative work, delete the known objects, and then mark the row deleted. A periodic reconciler can compare application records with objects under managed prefixes and classify discrepancies without treating the bucket listing as the source of authorization truth. Keep audit records long enough to demonstrate the deletion decision, but do not keep the deleted media inside those records. Compliance work gets messy when an audit log quietly becomes another content store.

Signed links are capabilities, not sessions

A signed URL grants whoever holds it the permitted operation until it expires. Keep it short-lived, restrict its method and object, and never log the full query string. The application should authorize every minting request against the tenant and asset record; obscurity of an object key is not authorization.

Cache behavior needs an explicit decision. If every viewer receives a distinct query string, a cache may fragment unless its cache-key policy is configured appropriately. If a shared cache serves private derivatives, verify that its key includes every dimension that separates authorized representations and that responses cannot cross tenants. For highly sensitive images, bypassing a shared cache may be worth the throughput trade-off. For a broad preview surface, a controlled delivery layer can reduce repeated object reads, but the authorization design must survive cache hits.

Use separate credentials and scopes for upload signing, derivative workers, cleanup workers, and read-link issuance.

Rotate them.

Alerts should cover growth in pending uploads, derivative queue age, verification failures, orphaned objects, deletion lag, and signed-link mint volume. A delivery dashboard without deletion lag is incomplete; retention failures can stay invisible while read latency looks healthy.

Signed links also need clock discipline. Keep signer hosts synchronized, use the shortest practical validity window for the workflow, and let clients request a fresh link through normal authorization rather than extending a leaked capability. Don't embed signed links in durable database fields or events. Store the object identity, then mint access at the edge of use.

Test throughput and cleanup as one system

Test with the object sizes and concurrency distribution the application expects, including an interrupted multipart upload, duplicate completion messages, repeated derivative jobs, deletion during derivation, and a read request immediately after deletion begins. The important metric is not application request latency alone. Track time to authorize an upload, client-to-storage throughput, time from upload verification to first usable thumbnail, retry volume, and bytes retained by lifecycle class.

Roll out retention rules in observation mode where the platform and process allow it: first classify candidate objects and compare them with database state, then enable deletion for a narrow prefix or cohort. Before broadening the rule, sample objects near each boundary and exercise restore or regeneration using the artifacts you intend to keep. Lifecycle configuration is infrastructure code and needs review, version history, and a rollback plan even though deleted bytes may not be recoverable.

What do we deliberately stop keeping? Abandoned uploads after their recovery window, replaceable intermediates once their successors are verified, and originals whose documented retention has ended. The failure cost is concrete: after cleanup, an expired original cannot rescue a corrupt or newly incompatible derivative. The compensating controls are checksum verification before promotion, idempotent generation, a tested grace period, and clear user-facing retention semantics. This pattern is not suitable when every historical source must remain reproducible forever; in that case, retain immutable originals and design the budget, access controls, and deletion exceptions around that obligation.

References