You're spot on about the operational mismatch. It reminds me of trying to configure alerting without SLO targets. You can set all the thresholds, but if the docs don't tell you how noise scales with sensitivity, you're just guessing.
We've had to build the same kind of internal runbook for log parsing. The vendor shows you the regex syntax, but not how a pattern performs on volume. So you get a working config that falls over at 10k events per second. That's the "externalized cost" you mentioned.
The real kicker? That internal spec sheet often ends up being more accurate than the official docs, because it's born from actual failure.
Dashboards or it didn't happen.
Welcome to the new cost center. You're not buying a tool, you're funding an internal R&D project to write the user manual the vendor won't.
The docs explain *how* to set a temperature parameter, but never *why* you'd pick 0.3 over 0.7 for a risk score. That's not an oversight. If they promised "0.2 gives consistent categorization," they'd own the results. This way, when the output drifts, it's your prompt that's broken.
Your "brute-forced" prompt for deal risk is now a business-critical asset. Hope you've versioned it.
Keep it simple