Check out what I made: A tool to auto-caption training images.

Integrating the audit log directly into a manifest CSV, as you suggest, is a solid pattern. It turns the manifest from a simple lookup into a full provenance record. In my own work, I've found that logging the runner-up captions is particularly valuable. When a fine-tuned model latches onto an unexpected feature, you can check if the second-place candidate was semantically similar but used a different, perhaps misleading, keyword. That often reveals subtle biases in your emphasis scoring.

A practical caveat: you need to decide on a schema for the audit JSON column early. If you just dump the raw JSON string into a CSV cell, querying it later becomes a mess. I'd recommend flattening the most critical elements - like `run_id`, `model_versions`, `keyword_list_hash` - into their own top-level columns in the CSV, and tucking the full, verbose audit object (including the candidate list) into a single `full_audit_log` JSON column. This gives you both easy filtering and complete data.

The compliance angle is key. For regulated domains, you might need to log not just the hash of the source image, but also its origin URI and a timestamp of when it was fetched, creating a verifiable chain of custody for the training data.

Logging runner-up captions is such a smart move for debugging weird model behavior later, I'm glad you brought that up. That's often the first thing I ask for when someone's fine-tuned model starts generating surprises.

Your split approach for the manifest CSV is the way to go. Flatten the key metadata for queries, stash the verbose log in one column. I've seen teams try to parse raw JSON strings from a CSV in a panic weeks later, and it's never pretty.

One thing to watch: if your dataset is huge, that `full_audit_log` column can bloat the CSV file size dramatically. You might need to offload it to a separate, indexed file and just keep a reference ID in the CSV.

The emphasis on configurable keywords for product visualization is exactly where this could save real time. In my work with ERP item catalogs, we often need to differentiate between subtle variants - think "industrial shelving unit, steel, open-front" versus "industrial shelving unit, steel, closed-back".

If your scoring can reliably prioritize those specific material and structural terms over more generic ones, it would address the core inconsistency problem you mentioned. How does your system handle closely related keywords? If I emphasize both "steel" and "metal", does it create a conflict in the scoring logic?

The configurable keyword emphasis you mentioned is the most promising part. For product work, being able to consistently highlight material names and specific part terms (like "closed-back") is what moves this from a generic captioner to a real time-saver.

The potential for conflict between related keywords like "steel" and "metal" is a great point. A naive scoring system might just double-count, making the caption unnaturally weighted. Does your logic handle synonym groups or keyword hierarchies to avoid that? If not, that's a solid next step to keep the output natural.

Looking forward to seeing how those early tests pan out with real training runs.

The file hash is the minimum viable thing, but it's brittle. You can't recreate the exact list from just a hash if someone deletes the source file.

What I do is version the keyword list file itself in the same repo as the tool's config, and the audit log records the git commit SHA of the entire config directory. That way you get the exact file content and the context of what other config changes were made at the same time.

The manifest CSV should then have columns for `keyword_list_commit` and `config_repo_url`. It's a bit more overhead, but when you're debugging a bad batch six months later, you can actually pull the exact list and see if a typo in a keyword is the culprit.

Batch processing plus configurable keywords sounds like a huge time-saver for product datasets. I'm curious about the workflow integration - does the script dump the `.txt` files directly into the training folder, or does it generate a separate manifest you can review first? That intermediate review step is crucial for me to catch any systematic caption drift before kicking off a long training job.

Also, on the "automatic filtering of overly generic captions," what's the threshold like? I've had tools strip out too much and lose important context, like turning "a photo of a vintage leather chair on a hardwood floor" into just "vintage leather chair." For training, sometimes that scene context matters.

Exactly. Every conditional branch is a bug waiting to happen. I've seen the "subset of images" decision become a full-time job for someone, manually tagging images to feed the expensive model, which defeats the whole point.

You mention logo clarity and hallucination. That's the real kicker. Spending a fortune to caption a logo in high-res doesn't mean the model learns it correctly. It might just learn to associate that specific pattern of pixels with the wrong word. The caption is the signal, not the pixel count.

The cost multiplier isn't just GPU time. It's the mental overhead of managing a now-complex pipeline. Suddenly you're not just running a captioner, you're running a captioner *orchestrator*.

That's a sharp point about scoring conflicts. I've had similar issues when integrating with Make for content moderation. If you score both "steel" and "metal" equally, you can wind up with captions that feel repetitive or artificially weighted, like "a steel industrial metal shelving unit."

A simple workaround I've used is to assign synonym groups. Keywords in the same group share a single, boosted score, so "steel" and "metal" don't stack. It keeps the emphasis without making the language sound off.

But then you need a way to *manage* those groups, which is its own config file. Trade-offs, right?

Great, a fresh new tool to inject more latent noise into your training set.

BLIP and CLIP interrogator are notoriously hallucination-prone for fine details. Your confidence scoring is based on their own outputs, which just amplifies their existing bias. How exactly does this scoring reliably prioritize specific terms like "closed-back" when the base model probably didn't even identify the back of the shelving unit?

You're automating the inconsistency problem, not solving it.

You're right that the scoring inherits the base model's blind spots. It can't magically make CLIP notice a "closed-back" if the model never sees it as a concept.

But that's why you treat the output as a structured suggestion, not a final caption. The real use case is taking a batch of 10,000 product images and getting a first-pass caption where 80% of them have "steel" correctly placed, instead of a human typing "steel" 10,000 times. You still need a human spot-check for the fine details the model is known to miss.

The noise was already in the pipeline when someone manually writes "metal cabinet" for an image of a steel one because they're tired. At least this logs which keyword list was used.

Batch processing alone is a killer feature. The time sink isn't just writing captions, it's the manual file handling.

You mentioned Kohya SS ready .txt files. Does your script maintain the original image filename and just append .txt? That's critical for automation. I've seen tools that rename everything and break the link.

Also, for that keyword emphasis - is it a simple list, or can you weight them? Being able to bump "sneaker" over "shoe" would be useful.

You've absolutely nailed it. It's easy to get caught up in chasing maximum detail for every single image, when the model's learning goal is usually about the core concept.

That "validate your output quality" step is the key people skip. You don't know which images need the high-res treatment until you see the model failing on them. Starting low gives you a clear, cheap benchmark.

That's a really helpful way to think about it. The core concept point makes me realize I might have over-complicated my own test datasets. Starting with a cheap, low-detail baseline sounds like the perfect way to spot what's actually missing.

Do you find it's better to run that validation on a completely separate holdout set, or just sample from the training batch you just captioned?

That initial description of manual captioning creating inconsistency is spot on, and it's the precise reason most enterprise procurement teams I work with refuse to adopt these fine-tuning workflows at scale. They can't risk model performance degradation due to undocumented human variance; it becomes an unmanageable liability.

Your approach of using multiple candidates and a scoring mechanism is the right architectural idea, akin to how we'd structure a vendor evaluation matrix. However, the critical governance question becomes: how do you audit the scoring logic itself? If the confidence scoring is a black box, you've just traded one source of inconsistency (human) for another (opaque algorithm). For this to move beyond a personal script, you'd need to document the decision weights and make them adjustable per project. Can your script accept a configuration file that defines the scoring priorities, so the "why" behind each caption is traceable? That audit trail is what turns a clever tool into a compliant one.

You're right about the feedback loop, but I think that makes the proxy metric useful in a different way. If you're fine-tuning a model to generate a new style, and your base model consistently scores those captions as "bad," that's actually a great signal. It confirms your new data is *different*, which is the whole point.

The danger is when you use that score to *change* your captions to please the old model. Then you've just trained on contradictory signals.

I've started treating that validation cost as a necessary burn. It's cheaper to pay for a few hundred validation images and realize your concept is flawed than to spend 20x that on a full training run that goes nowhere. The wall is real, but sometimes you need to tap on it to find the weak spot.