Totally agree on the hybrid approach. The Jenkins metadata is your primary source, but you have to assume it's incomplete. We used a similar method - crawling the artifact directory's modtimes against the build timestamps in the database to find the "ghost" artifacts. It caught old SNAPSHOT jars from manual deploys that were still linked from ancient wiki pages.
The append-only ledger for the map is a lifesaver. We structured ours as individual CSV logs per job run, which then got merged into a central lookup service. It prevented so many merge conflicts and gave us a clear audit trail when a redirect was missing.
Your last point on scoping the list operation is so true. Starting with a full recursive list on a huge bucket is a rookie move that burns time and money. Metadata first, then a targeted, paginated crawl on just the directories you actually need.
measure twice, ship once
That audit trail from the mergeable logs is a huge win. We used a similar CSV approach, but we quickly realized we needed to embed a checksum of the artifact itself into the mapping record. When we later moved from one object store to another, we could verify the transfer's integrity by recalculating checksums at the destination and matching them back to the ledger. It turned a simple redirect map into a full chain of custody.
The modtime-to-build-timestamp correlation is smart for finding ghosts, but it assumes the filesystem timestamps are trustworthy. In our case, some bulk operations had reset timestamps en masse. We had to fall back to correlating artifact naming patterns against the actual build numbers in the CI database as a secondary check.
Measure twice, spend once
The legal retention schedule is a nice theory. In practice, that document is usually a PDF from 2018 that says "comply with all applicable laws." Good luck deriving a file deletion policy from that.
You'll still need the inventory. The trick is generating it without paying for a full storage crawl. If you can't query access logs or build metadata first, you're already in trouble.
SQL is enough