I've been trying to get Traceloop to track some project-specific metrics, like the number of function calls to a particular internal tool or the average score from a custom validation step in my chain. The standard dashboard is great for tokens and cost, but the documentation on extending it feels sparse.
Has anyone here successfully set up custom metric tracking? I'm looking for practical examples: did you use the SDK's span attributes, the evaluation features, or something else entirely? A walkthrough of your configuration would be incredibly helpful for the community.
I'm particularly interested in how you then surface those metrics in the UI or export them for reporting. Let's keep the discussion focused on the technical how-to and avoid general commentary on the product.
Remember the rules
I ran into this too, trying to count specific tool uses. I ended up using span attributes to attach counts, like a "tool_xyz_calls" tag. But then I got stuck - I couldn't figure out how to build a dashboard for those tagged counts. The UI doesn't seem to group them automatically.
Did anyone find a way to actually chart these custom numbers? Or is the only real option to export all the data and crunch it in a spreadsheet?
I've been using the evaluations feature for something similar. It felt more aligned with tracking scores than trying to misuse span attributes.
In my setup, I created a custom evaluator that runs after a validation step. It pushes a numeric score to Traceloop. The config looks something like this:
```python
traceloop.evaluate(
name="validation_score",
data={"score": calculated_score},
type="validation"
)
```
The tricky part is, you're right, the UI doesn't automatically chart it. I had to use the "Custom Dashboards" beta feature in the project settings to build a view for it. It's a bit hidden 😅
Has anyone else tried the evaluations path, and did you find a way to export that data via their API? I'd love to pipe it into our own monitoring.