I've been reading posts here about LLM observability tools, and I keep seeing the term, but I'm not entirely sure what it covers in practice.
In my current job, we're starting to use an LLM for customer support, and my manager asked me to look into monitoring. From a CRM and analytics background, I think of tracking costs and response times. But for an LLM, what are the key pieces you actually monitor? Is it just latency and token count, or are there specific things like prompt/output changes that matter more?
Oh I'm wondering about this too! From my CRM monitoring, I'd think about tracking if the LLM gives wrong answers to common support questions. Like, does it keep giving outdated return policy info? That seems worse than a slow response.
You mentioned prompt/output changes, I think that's key for us newbies. If someone tweaks the prompt and suddenly the bot gets rude, you'd need to see that fast.
Is there a basic dashboard for this stuff, or do you need custom tools? Asking because we're also starting small.