That callback example is the least of it. The real pain comes later when you try to query that data. You'll spend more time figuring out why your custom metric is bucketed under their "proprietary insights" panel than you ever would have spent writing a few lines of Python to log directly to a time-series database. The leak isn't just in the SDK, it's in the entire premise that your internal audit trail should fit into their generic LLM evaluation schema. For checking policy doc citations, you don't need a schema. You need a grep.
Trust but verify.
You hit on the exact frustration. That "reading their source" moment is where the promised productivity gain vanishes.
We did something similar for tagging user journey stages from our CRM. The vendor SDK made it seem trivial until we needed the attribute to propagate to child spans for a cost breakdown. Suddenly we're patching SDK internals just to get our own data through.
Your OTEL collector suggestion is the pragmatic escape hatch. It's boring, but it's stable. The custom exporters let you shape the data for your alerts without fighting a black box every quarter.
Keep it simple.
> the pragmatic escape hatch
So, if I'm getting this right, the OTEL collector just sends the raw data somewhere you control, like a database you already manage? Then you build alerts off that.
That sounds way less magical, but maybe that's the point. For our small team, adding a whole new query language on top of a vendor UI is a lot. But we could probably handle writing a simple cron job to check a Postgres table.
Is the main benefit that you stop caring when they change their own UI?
CloudNewbie