I'm neck-deep in a project generating product mockups with DALL-E 3 via API. We're talking 5k+ images a month, each in multiple variants (square, portrait, etc.). My current S3 bucket is becoming a mess.
I need a system that's not just a dumping ground. The core requirements:
* **Traceability:** Must be able to link an image back to the exact prompt and generation parameters.
* **Versioning:** We often iterate on a base concept; need to keep those lineages clear.
* **Cost-effective storage:** Most images are accessed rarely after the first week.
* **Metadata querying:** Being able to find "all images from campaign X" or "all v2 of concept Y" without manual tagging.
I've sketched out a couple of approaches, but I want the hard-won lessons.
**Option A: S3 with a sidecar database**
Store raw images in S3, organized in a path like `s3://bucket/{project_id}/{generation_id}/{variant}.png`. All metadata (prompt, model params, cost, timestamp, S3 URI) goes into a DynamoDB or PostgreSQL row, keyed by `generation_id`. This feels clean but adds operational overhead.
**Option B: S3 metadata and intelligent prefixing**
Lean heavily on S3 object tags and user-defined metadata. Use a strict prefix convention: `{source_model}/{date}/{concept_hash}/{iteration}.png`. Query with S3 Inventory or Athena if needed. Simpler, but I'm skeptical about complex queries later.
**Option B: S3 metadata and intelligent prefixing**
Lean heavily on S3 object tags and user-defined metadata. Use a strict prefix convention: `{source_model}/{date}/{concept_hash}/{iteration}.png`. Query with S3 Inventory or Athena if needed. Simpler, but I'm skeptical about complex queries later.
**Option C: Something else entirely?** Heard of people using vector databases for image embeddings to find similar gens, but that seems like overkill for my core audit and retrieval needs.
Biggest pitfalls so far:
* Not planning for deletion or archival lifecycle policies from day one.
* Letting filenames become meaningless UUIDs with no context.
What's working for those of you at scale? Is the database route worth the extra setup? Specifically interested in AWS-native patterns and their cost profiles.