Hey everyone! I've been experimenting with Traceloop for monitoring some Lambda functions. I was curious how its "quality" and "cost" scores actually line up with a real human looking at the outputs.
So I built a simple dashboard that pulls Traceloop's evaluation metrics (like `trace.quality.score` and `trace.cost.score`) for a batch of traces and compares them to scores I manually gave based on whether the output was actually correct/useful.
Initial finding is... interesting! For my simple use case, Traceloop's quality score seems to correlate pretty well when the LLM output is totally wrong or hallucinates. But for more subtle "kinda correct but not great" answers, the correlation isn't as strong.
Here's the basic query I used to pull the data from their BigQuery integration:
```sql
SELECT
trace_id,
JSON_VALUE(attributes, '$.trace.quality.score') as tlp_quality_score,
JSON_VALUE(attributes, '$.trace.cost.score') as tlp_cost_score
FROM
`my_project.traceloop.traces`
WHERE
span_name = "my_llm_chain"
```
Has anyone else done a similar comparison? I'm wondering if I should be using custom evaluators or if the built-in scores are good enough for basic health checks. Also, how much do you trust the cost score for spotting wasteful patterns?
Your finding about correlation dropping on subtle outputs is the key takeaway. Automated scores are proxies, not replacements. For basic health checks on clear failures, they're fine. But if "kinda correct but not great" matters for your use case, you'll need custom evaluators targeting those specific failure modes. I'd be interested to see if the correlation improves when you break your human rating into separate dimensions like factual accuracy vs. completeness vs. helpfulness, and check each against Traceloop's sub-scores.
—AF
That's a good idea about breaking the rating down. Traceloop's quality score is a single number, so it's probably averaging things we'd separate.
But how do you objectively define dimensions like "helpfulness" for a custom evaluator? Seems subjective and expensive to scale.