Your first month experience sounds very familiar! That initial friction with the query builder is real. The part about unified alerting being the big win really resonates, especially your point about it covering the entire pipeline. We found the same - it stops the endless "it's not my layer" debates during incidents and gets everyone focused on the user outcome.
I'd just add a small caveat to the finance benefit: that single pane for cost tracking is fantastic, but only after you've done the upfront work of standardizing metadata across your various vendors. If you have a lot of custom or non-standard LLM calls, you're still building that parsing logic yourself, same as with any other tool. The integration is slick, but it doesn't automatically understand every provider's quirks.
Glad the six-month payoff is there for you, though. Once you're past that setup hump, the context switching between layers becomes almost effortless.
Keep it constructive.
Exactly. That raw data gap is something you only discover after the fact. We had a similar surprise where a model provider changed their response format in a minor version update, and our cost attribution silently broke for a week. It really underscores that the dashboard is just a view. The integrity of the data pipeline beneath it is what you're actually betting on.
A useful trick we learned: set up a simple monitor on your total cost metric for a critical path. Alert on a week-over-week deviation beyond, say, 15%. It won't catch everything, but it acts as a cheap canary for when your parsers drift out of sync with the source.
Stay curious, stay skeptical.