Just built a dashboard comparing Traceloop metrics to human rating scores.

cloud_infra_newbie · 2026-08-07T04:10:59Z

Hey everyone! I've been experimenting with Traceloop for monitoring some Lambda functions. I was curious how its "quality" and "cost" scores actually line up with a real human looking at the outputs. So I built a simple dashboard that pulls Traceloop's evaluation metrics (like `trace.quality.score` and `trace.cost.score`) for a batch of traces and compares them to scores I manually gave based on whether the output was actually correct/useful. Initial finding is... interesting! For my simple use case, Traceloop's quality score seems to correlate pretty well when the LLM output is totally wrong or hallucinates. But for more subtle "kinda correct but not great" answers, the correlation isn't as strong. Here's the basic query I used to pull the data from their BigQuery integration: ```sql SELECT trace_id, JSON_VALUE(attributes, '$.trace.quality.score') as tlp_quality_score, JSON_VALUE(attributes, '$.trace.cost.score') as tlp_cost_score FROM `my_project.traceloop.traces` WHERE span_name = "my_llm_chain" ``` Has anyone else done a similar comparison? I'm wondering if I should be using custom evaluators or if the built-in scores are good enough for basic health checks. Also, how much do you trust the cost score for spotting wasteful patterns?

Page 3 / 3 Prev

Traceloop Reviews

Last Post by Consultant Carl 37 minutes ago

31 Posts

29 Users

0 Reactions

3 Views

RSS

Consultant Carl

(@consultant_carl_42_v2)

Reputable Member

Joined: 4 months ago

Posts: 222

08/08/2026 2:33 pm

Exactly. Your approach of pairing a simple, rule-based proxy with the generic score is the practical middle ground. It mirrors a procurement principle we use: define your "must-have" requirements as binary checks, then use a weighted scorecard for everything else.

One caveat: that entity-check rule only works if you have a reliable way to extract those key terms from the user's question in the first place. We've seen teams spend more time tuning their extraction logic than they save on evaluations. So the maintenance cost you mention can sneak in earlier than expected.

null

ReplyQuote

Page 3 / 3 Prev