Hey everyone! 👋 I've been lurking here for a while, learning a ton from all your posts. I'm pretty new to this whole B2B software evaluation process, so thanks for all the shared knowledge.
We're a mid-sized startup, and our engineering team (15 people) recently started using OpenAI and Anthropic APIs in a few of our features. It started with just one dev's prototype, but now it's in three different services. Our CTO asked me to figure out some way to "see what's happening" with all these LLM calls—especially since costs were starting to pop up on the bill and we had a weird latency spike last month that was hard to debug.
Honestly, I was overwhelmed at first. I didn't know where to start. We needed something that could track costs per feature, see latency, and catch errors. Most importantly, it had to be something the whole engineering team would actually *use* without a huge learning curve.
Here's what we ended up doing, in a pretty basic sequence:
1. We first tried to build some basic logging ourselves, but it got messy fast across different services.
2. I looked at a few dedicated tools (some names I see a lot here). We picked one that seemed to have the smallest "time to value" for our stack (mostly Node.js and Python).
3. The rollout was the scariest part. I was worried engineers would see it as extra overhead. We started by just instrumenting one non-critical service, shared the dashboard in our team Slack, and showed how we could finally attribute a cost spike to a specific, overly verbose prompt.
The win was when another team used the tracing to debug a timeout issue in under an hour—something that would have taken days of guesswork before. We're not experts, but now we at least have a basic map of what's going on.
I'm curious—for those of you who've done this, how did you get initial buy-in from your engineers? Was it the cost tracking or the debugging that sold it?
Ah, the inevitable journey from "just log it ourselves" to "let's buy a magic box." I can already guess which tool you picked, and while it'll give you pretty dashboards, wait until you try to trace a single user request that fan-outs to multiple LLM calls across those three services.
That vendor's auto-instrumentation is notoriously leaky in anything but a trivial monolith. You'll get costs and latency averages, but good luck understanding why request #123456 had a 20-second tail latency when the p99 looks fine. The real observability happens when you can tie those LLM spans to your own business logic in the same trace. Most of these tools treat your app as a black box that makes API calls.
Did you consider just exporting OpenTelemetry spans from your existing instrumentation and adding LLM semantic conventions? It's a gnarlier setup, but it actually works when things go sideways.