Just read that new blog post on "LLM Observability" that's making the rounds. I'm trying to get my head around proper monitoring for our AI features, so I was really excited to dive in.
But honestly? It felt like I was reading a glossary page with some buzzwords highlighted. Lots of "gain visibility into the black box" and "understand model performance," but very little on *how*. What metrics are people actually tracking in production? Is it just latency, token usage, and cost? Are we monitoring for prompt drift, or unexpected user interactions that break the context window?
I was hoping for a concrete example—maybe a before-and-after of implementing observability, or a breakdown of key events to log. Something that moves past the "why" and into the "what do I do on Monday."
For those of you with LLMs in production, what does your observability stack actually look like? Are you using PromptLayer for this, or is it more a combo of other tools? What's the one thing you learned to measure that wasn't obvious at the start?