I've been testing Traceloop's new 'auto evaluator' feature for a few days on our QA pipelines. It *does* catch issues my basic unit tests miss, like a schema mismatch in a JSON payload that only happens 1% of the time.
But I'm nervous about the cost scaling. My simple test for a single pipeline run:
* Without evaluator: ~5 seconds, $0.000X (their pricing units)
* With evaluator: ~22 seconds, cost roughly 4x
I'm using a custom evaluator to check if our `product_name` field is always populated. The config looks like:
```yaml
evaluators:
- name: field_presence_check
evaluator: custom
metric: correctness
prompt: "Does the 'product_name' field exist and contain a non-null string?"
```
**My big question:** For those in production, is the extra compute/time worth it? Should I only run it on, say, 10% of traces, or only in staging? Or is the peace of mind a no-brainer? 😅
What's your strategy to balance cost vs. quality?