That's an excellent point about idempotence that I hadn't fully considered. You're right that as we add to the dictionary and rerun, we need a way to avoid reprocessing the same records. Our current script is a one-time bulk run, but the plan to handle the log file entries will create that exact problem. I think we'd need to add a simple "last_migration_date" flag on the records in our source database, or maybe a sentinel value in Claw's metadata field, to mark what's already been processed.
On the granularity question, your mention of storing the original tag as metadata is something I'll bring up with the team. We're collapsing them for the main classification field, but keeping that audit trail would let us adjust later if Claw's categories change. It feels like a small addition that could save a lot of headache down the line. Have you found that storing the original tag causes any performance issues in queries, or is it generally negligible?
Exactly, that metadata flag is the right move. We went with a `_migrated_to_claw_v1` boolean column on our source records. It's simple and prevents any reprocessing headaches. Your idea for a sentinel in Claw's metadata also works great if you can't modify the source.
On performance, storing the original tag as a supplementary field hasn't been an issue for us. The overhead is negligible, and it's a lifesaver for auditability. We just added an indexed JSON blob field in Claw for `original_metadata`. The only real cost is extra storage, which was trivial compared to the peace of mind when we had to re-evaluate some ambiguous tags later.
What's your team thinking for the actual update mechanism? Are you doing a single batch SQL update with the flag, or something more event-driven?
— francesc