Hi everyone! I’ve been lurking here for a bit, learning so much from all your discussions on evaluation. 😊 I work for a small e-commerce company, and we recently launched an LLM-powered live chat agent. We’ve been wrestling with the classic trade-off: the agent can give really thoughtful, accurate answers (but slower), or it can give quick, sometimes generic replies.
Our product manager wanted a single metric to track “overall performance,” so I just built a combined score that balances response quality and latency. I thought I’d share what I came up with—I’d love to get your feedback, especially since I’m new to this.
I’m using our existing tools: a simple quality score (1-5, rated by human reviewers on correctness and helpfulness) and our system’s response time in seconds. The trick was normalizing them and giving them weights. We decided that for our use case, quality is a bit more important than speed, but not by a huge margin.
Here’s the basic formula I implemented:
`combined_score = (0.6 * normalized_quality) + (0.4 * normalized_speed)`
To normalize, I set a target for each. For quality, a score of 5 is our “perfect” (so `normalized_quality = raw_score / 5`). For speed, we defined an “ideal” response time of 2 seconds and a “maximum acceptable” of 10 seconds. Any response faster than 2 seconds gets a 1.0 for speed, anything over 10 seconds gets a 0.0, and anything in between is scaled linearly.
It’s super simple, but it’s already giving us a much clearer picture than looking at two separate dashboards. We can see at a glance if tuning for better quality is tanking our speed score too much, or vice versa. Has anyone else tried something similar? I’m curious about different ways to handle the normalization or if there are pitfalls I haven’t considered.