Hey everyone,
I've been diving deep into Traceloop for monitoring our LLM calls and traces, and it's been a game-changer for spotting inefficiencies in production. However, I've hit a snag with billing that I'm sure others are facing.
Our CI/CD pipeline runs a ton of integration tests that trigger LLM calls, and our developers are constantly experimenting in their feature branches. All of this is generating traces that get sent to Traceloop, and I'm pretty sure it's inflating our usage count (and the bill!). We're on a plan with monthly trace limits, so this "noise" isn't just a data cleanliness issue—it's a cost issue.
Has anyone figured out a clean way to filter out traces from non-production environments *before* they count towards billing? I'm thinking along a few lines:
* **SDK-level configuration:** Is there a way to conditionally disable the Traceloop exporter based on an environment variable (like `NODE_ENV !== 'production'`)? I'd rather not have to maintain separate builds.
* **Proxy or collector rules:** Can we set up the OpenTelemetry Collector to drop or sample traces from certain environments? What would that configuration look like?
* **Tagging and filtering:** Maybe we tag all our dev traces with an `environment=dev` attribute and then somehow ensure those are excluded from the billed volume?
I'm especially interested in solutions that don't require manually toggling things on/off for developers. It should be automatic based on the deployment target.
What's working for you all? I'd love to compare notes and find the most elegant, "set-it-and-forget-it" approach.
Keep automating!
Keep automating!
You can absolutely disable the exporter based on environment. Set the `TRACELOOP_DISABLED` env var to `true` in your dev/test environments. That's the cleanest SDK-level fix.
But your proxy idea is better for a large setup. Run a dev-specific OpenTelemetry Collector that either doesn't forward to Traceloop, or uses a sampler to drop 100% of traces. Keep your prod collector config separate. It's a bit more infra, but it keeps your code clean and prevents any accidental leaks.
Have you checked if Traceloop's own API supports setting a `environment` tag on the span? If they bill based on ingested traces, tagging them as "dev" and then filtering at their end might be possible. That's usually a vendor-specific feature though.
Integration is not a project, it's a lifestyle.
The environment variable approach is elegant for simple cases, but it's brittle in a containerized CI environment where variables can be inherited or misconfigured. A single mis-set job can flood production with dev traces, which defeats the cost isolation goal.
Your point about the proxy and a separate collector configuration is the correct architectural pattern. It moves the decision logic out of the application code and into the observability infrastructure, where it belongs. For anyone implementing this, I'd recommend coupling it with a sampling strategy at the collector level - perhaps head-based sampling for dev and tail-based for prod - to further manage volume and cost, even within the filtered stream.
I'm not aware of Traceloop billing filtering on tags, but that's a vendor question. Most observability platforms bill on ingested data volume, so filtering before egress is the only reliable cost control.
Data doesn't lie, but folks sometimes do.