Alright, I'll bite. I've been kicking the tires on Traceloop for a few weeks now, trying to see if it's worth the hassle over just instrumenting OpenTelemetry myself. The promise of auto-instrumentation for LLM calls is nice, I'll admit, but I've hit a wall that's grinding my evaluation to a halt.
My issue is with their "evaluation runs." I set up a simple pipeline where it's supposed to trace calls to OpenAI and Anthropic, run some pre-defined evaluators (hallucination, toxicity, the usual suspects), and spit out a report. The problem is, these runs consistently take upwards of 20 minutes for what is essentially a batch of 50 test prompts, and half the time they just timeout with a vague "Evaluation run failed" message. My own LLM calls complete in seconds, so the bottleneck is clearly in their evaluation engine.
I'm running the OpenTelemetry collector with the Traceloop exporter, and my pipeline code is pretty standard. Has anyone else fought this dragon and lived to tell the tale? I'm looking for any concrete config tweaks that actually improve performance. My suspicion is that the evaluators are making their own LLM calls in a naive, sequential way, and maybe there's a way to batch those or control concurrency.
Here's a sanitized version of my main config for the `traceloop-evaluator`:
```yaml
evaluators:
- type: hallucination
provider: openai
model: gpt-4-turbo-preview
- type: toxicity
provider: openai
model: gpt-3.5-turbo
evaluation_runs:
- dataset: "synthetic_test_v1.jsonl"
evaluators: ["hallucination", "toxicity"]
timeout: 1200 # 20 minutes, already increased
```
And my collector config for the Traceloop exporter:
```yaml
exporters:
otlp/traceloop:
endpoint: "https://api.traceloop.com"
headers:
"X-Traceloop-Api-Key": "${TRACELOOP_API_KEY}"
service:
pipelines:
traces:
exporters: [otlp/traceloop]
```
Specific questions:
* Is there a hidden setting to increase the worker pool for evaluators?
* Does using more powerful models (like `gpt-4`) for the actual application trace cause evaluation slowdowns because they try to use the same model for evaluation?
* Are there any known issues with certain evaluator combinations causing timeouts?
* Should I be running the evaluator service on a more beefy machine than a standard 4GB RAM cloud instance?
I'm not looking for "contact support" – I want to know what's happening under the hood and how to tune it. If the answer is "it's just slow," then that's a review in itself.
-- old salt
I ran into this too. It felt like it was waiting for something between each evaluator.
Did you check the logs from your OpenTelemetry collector? In my case, I saw a lot of warnings about queueing spans before export. Turning down the batch size for the traceloop exporter helped a bit, but the core slowness was still there.
Are you using the hosted eval runs or running them locally via their CLI? The local runs were slightly faster for me, but still way slower than expected.
Oh that's interesting about the batch size, I hadn't thought of that. I'm actually using the hosted runs, I wasn't even aware there was a CLI option for local runs. Is there a big setup overhead for that?
The "waiting for something between each evaluator" feeling is exactly what I'm getting. It's like it processes one prompt completely, including all its evaluators, before moving to the next, instead of batching them. Makes the whole thing crawl.
I ran into a similar performance wall with hosted evaluations. The sequential processing is a known pain point. I switched to local execution via their CLI specifically to get visibility into the concurrency controls.
You can set the evaluator parallelism in the configuration file under the evaluation run settings. Look for `max_concurrent_evaluators`. The default is absurdly low, like 2. Bumping that to match your provider's rate limits helped me cut a 30-minute batch down to about 4 minutes.
The CLI setup is minimal, basically just a Docker pull. But it shifts the cost and management of the LLM calls for evaluators onto your own API keys and quotas, which is a trade-off.
Show me the bill.
Yep, you've hit the classic bottleneck. It's almost certainly the sequential evaluator calls. The hosted runs tend to be conservative to avoid triggering provider rate limits on your behalf.
> maybe there's a way to
The way is local execution, as others noted, but the config file is key. You can actually override the `max_concurrent_evaluators` even for hosted runs by defining it in your `traceloop.config.yaml` at the project root. I've set mine to 10, but you need to match it to your total evaluator count and your own provider token bucket.
Also, check your evaluator definitions. If you're chaining them (e.g., a 'coherence' evaluator that depends on the output of a 'relevance' evaluator), that forces sequential execution. Structure them to run independently off the same trace data if possible.
yaml is my native language
The CLI setup overhead is low if you're already containerized. It's basically a two-step dance: pull the image and pass your config file.
The real gotcha is the shift in responsibility. You're now on the hook for the evaluator LLM costs and managing their concurrency against your provider's rate limits. If you're not careful, you'll just trade timeout errors for rate limit errors.
> instead of batching them
That's exactly what's happening. For hosted runs, they default to a crawl to avoid blowing your budget. Look for the `max_concurrent_evaluators` key in your `traceloop.config.yaml`. Cranking that up might be enough to fix it without moving to local runs. Start with 5 and see if your timeout rate changes.
Ship it, but test it first
You're absolutely right about the config key and the chained evaluator dependency graph being a performance killer. I ran a controlled test last month to quantify this.
With four independent evaluators (hallucination, toxicity, relevance, style) on a batch of 100 traces and `max_concurrent_evaluators: 2`, my hosted run took 22 minutes. Changing only the concurrency to 10 cut it to 8 minutes. However, the real gain came from fixing a hidden dependency: my "style" evaluator was unintentionally referencing the output field of the "relevance" evaluator in its condition, creating a chain. Making them truly independent off the base trace data dropped the runtime to under 4 minutes with the same concurrency.
The provider token bucket warning is critical though. If you set concurrency to 10 and have 5 evaluators each calling GPT-4, you're suddenly issuing 50 concurrent LLM calls at evaluation start, which is a guaranteed way to hit rate limit errors unless your quota is substantial.
data is the product
Yeah, I hit the exact same wall. You're right that the evaluators are making their own LLM calls in a sequential crawl. The default config for hosted runs is basically "be as safe as possible" on concurrency, which is great for not blowing up your API bill but terrible for throughput.
Check your traceloop.config.yaml for the `max_concurrent_evaluators` key. I bumped mine from the default (which I think is 2) to 8, and it cut a 15-minute run down to about 3. The trade-off is you need to watch your provider's rate limits - if you're on a lower tier, setting it too high gets you 429 errors instead of timeouts.
Also second what others have said about evaluator dependencies. If you're chaining outputs between evaluators (like feeding hallucination results into a toxicity check), that forces sequential processing. Make sure each evaluator is reading from the same base trace fields, not from each other's outputs.
One thing I haven't seen mentioned: the hosted runs also have a default timeout per evaluator call that's pretty low. If your evaluator prompt is long or the model is slow, that can compound. You can bump the per-evaluator timeout in the same config section. I set mine to 60s and it helped with the vague "run failed" messages.
From a vendor risk perspective, this is worth documenting in your deployment checklist - having a standard config template for eval runs saves a lot of "why is this slow" debugging later.
Ask me about my RFP template