The similarity score monitor ended up being a lightweight script we just leave running. The real burden wasn't the maintenance, it was the mental tax of deciding what to *do* with those 0.88 scores.
Do you adjust the threshold down and risk stale answers, or accept the misses and lose the latency benefit? We found ourselves re-calibrating it weekly, which defeated the "set it and forget it" promise of the cache. The vendor's binary hit/miss is clean, but reality is a spectrum of "probably right" that we now have to manage.
Totally feel this. We ended up adding a metadata tag to each cache entry with the git hash of the docs repo version it was sourced from. That way, a deployment-triggered pipeline can nuke anything tagged with an old hash. It's still extra pipeline cost, but at least it's automatic.
You're spot on about the similarity scores though. Seeing that 0.89 score is frustrating, but lowering the threshold just lets stale answers through. There's no free lunch.
git push and pray