Just started monitoring my LLM calls last month. I’ve been testing a few observability dashboards and got overwhelmed by all the charts.
From my 14-day trials, I’ve learned to ignore the noise and focus on three core metrics that actually changed my decisions:
- **Latency per token (or per call)**: Not just total time. Breaking it down showed me when my vector DB or a slow model was the bottleneck. Saved 40% on response time by spotting one slow integration.
- **Cost per user/feature**: Attribution is key. I track which endpoints or features drive the most LLM spend. Shocked to find a simple chat widget was 70% of my OpenAI bill.
- **Token usage vs. output quality**: I log token counts and pair them with a simple accuracy/helpfulness score (1-5). Spotted when longer prompts didn't improve answers—just cost more.
What do you veterans track? Are there any "health" metrics you watch that aren't obvious?
Trial number 47 this year.