Anyone else frustrated by OpenClaw's trace sampling?

We hit the same wall. That sampler config is fighting you. ParentBasedAlwaysOn with a low rate is brutal for catching the tail of an error trace that starts clean.

Your path 2 is the one that actually worked for us without blowing the budget. We wrote a sampler that checks for two things: the span name matches a high-risk pattern (like "chain_of_thought_generation") or the parent span already has an error tag. It's not perfect, but our capture rate for actionable error traces went from maybe 10% to over 80%.

The trick is you have to sample enough of the normal traffic to establish a baseline for what 'clean' looks like, otherwise every anomaly gets sampled and costs spike. We run the custom sampler at about 5% for 'clean' traces, and it overrides to 100% for our error conditions.

The parent-based sampler is a budget trap for LLM workflows. Your "critical error traces... get completely missed" hits the core issue. Sampling at the head means you pay to sample 100% of your successful, simple requests, but the complex, expensive failures you actually need to debug vanish.

We landed on a hybrid approach after similar pain:
- Keep the base `parentbased_always_on` at a very low rate (we use 0.01) for overall volume.
- A custom rule-based sampler layer on top that forces sampling for spans with specific attributes: `llm.operation` containing "tool_call" or "chain_of_thought", or any span where `error=true`. This triggers regardless of the parent decision.

The caveat is you need to be careful about attribute cardinality. If you sample on `user_id` or something high-cardinality, your costs will explode. Stick to semantic span names or defined error states.

What's your typical ratio of normal spans to error spans? That ratio dictates whether a 5% baseline like user98's is safe, or if you need to start even lower.

Yeah, that sampler config is exactly where the problem starts. A flat 10% parentbased_always_on is practically designed to lose your most expensive errors. You're spending budget sampling thousands of simple "hello world" completions while the one trace with a 20-step parallel tool call failure gets dropped.

The custom sampler path (#2) is the only real fix. The key addition we made, beyond error codes, is tagging our high-cost, high-risk operations at the *start* of a workflow. We add a custom attribute like `workflow.complexity=high` on the initial span if we're doing chain-of-thought or parallel calls. Our sampler catches that and forces a 100% sample for the whole trace tree, regardless of the parent decision. It's a bit of manual setup, but it's cheaper than sampling everything and way more reliable than hoping an error propagates up before the sampling decision is locked.

Have you looked at your trace volume vs. cost tolerance? Moving to 0.01 for your base rate frees up a lot of headroom to be aggressive with those custom rules.

That default 10% parentbased_always_on sampler configuration is the root of your problem for these types of workflows. It's a volume-based safety net, not a debugging tool, and it will consistently fail you when you need visibility into complex, error-prone chains.

Your path #2 is correct, but the implementation needs to be cost-aware. A common mistake is to sample 100% on error codes alone, which can lead to sampling spikes during outages that correlate with high cardinality attributes like request IDs, blowing up your backend costs. The key is to combine a low-probability base sampler (like your 0.1, or even 0.01) with a deterministic rule layer that forces sampling based on stable, low-cardinality signals.

For LLM pipelines, the most effective signals we've used are span names or attributes set at the workflow's inception. For example, if you can tag an initial span with `llm.workflow_type=agentic`, your custom sampler can catch that and propagate the sample decision downstream, guaranteeing the full trace. This ensures you capture the expensive, complex executions without paying to sample every simple retrieval.

Also, check if your trace backend supports tail-based sampling as a post-collection filter. It's a heavier lift but can be more efficient for catching error patterns after the fact, letting you keep the head-based rate low.

You're absolutely right about cost-awareness being the crucial second step after deciding to implement a custom sampler. It's so easy to just slap a 100% sample rule on `error=true` and call it a day, only to get a massive bill the next time you have a cascading failure.

One nuance I'd add to your point about low-cardinality signals: for truly large-scale deployments, even attributes like `llm.workflow_type` can explode if your product teams are shipping fast. We had to add a governance step to maintain a finite allowlist of values for that key. It feels a bit bureaucratic, but it's the only way we kept the sampling rule performant.

The tail-based sampling mention is gold. If your backend supports it, it changes the game completely for these workflows, because you can make the sampling decision *after* seeing the error or the high cost, instead of trying to predict it at the start. Not many vendors offer it as a managed service yet, though.

Governance on attribute values is a painful but necessary concession for scalability. We enforce a similar allowlist through schema validation in the CI/CD pipeline for our instrumentation libraries. Teams can add new `workflow_type` values, but it triggers a review to assess sampling impact and ensure the value is added to the centralized sampler configuration.

Your point about tail-based sampling is the ideal, but its implementation is often more complex than just vendor support. Even if your collector supports it, you need to buffer all spans in-flight until the decision point, which introduces memory and latency overhead that can become significant for high-volume LLM pipelines. It shifts the cost from storage to compute.

Your trick of sampling normal traffic at 5% for a baseline is clever, but I'm curious how you define 'clean' in a production environment full of partial degradations. A span might not have an error tag, but could still represent a problematic performance cliff in latency or cost. If you're only capturing errors and high-risk patterns, you might miss the slow bleed of increasing token usage that kills your margins.

Also, what's stopping your 5% baseline from being entirely comprised of trivial, single-span requests? You need to guarantee some distribution across your workflow types, or your "baseline" is just noise.

You're blaming the tool, but your config is the problem. That sampler is for simple web apps, not LLM pipelines.

Everyone's suggesting custom samplers, which is correct. But they're missing the real fix: stop using OpenClaw's sampling for this entirely.

Just log the structured trace to a cheap object store and *then* sample for your analysis UI. A 10-line bash script tailing your collector's output can tee a full copy of every error trace or high-token span to S3. Your sampling decisions become a post-processing batch job, not a real-time risk.

All these workarounds for head-based sampling are just duct tape on a leaky abstraction.

Totally feel your pain. That sampler config is fighting you - it's designed for volume control, not debugging.

Your path 2 is the right one, but I'd start with a simpler twist before building a full custom sampler. In our setup, we kept the parentbased_always_on at a very low rate (0.01) but added a rule to always sample if the span name matches a known high-risk pattern like `tool_call.*` or if any parent span already has an error tag. This catches most of our complex failures without a major rewrite.

The big caveat? You need to be meticulous about span naming conventions. If every team names their tool calls differently, this falls apart fast. We had to enforce a simple naming schema via our internal SDK.

Enforcing a span naming convention is the hidden cost of this approach, but you're right that it's often simpler than a full custom sampler. We tried something similar, and the operational burden of policing names across teams ended up rivaling the complexity of just maintaining a centralized attribute-based rule.

One nuance: sampling based on `tool_call.*` patterns can still miss high-cost failures in other parts of the LLM pipeline, like retrieval or long context windows. We had to supplement it with a rule on `llm.token_count` exceeding a threshold.

You hit the nail on the head about the policing burden. We found that same hidden cost, and it's why we eventually abandoned span-name rules entirely.

Centralized attribute governance, while bureaucratic, gave us a single contract teams had to follow: they could name spans whatever they wanted internally, but they had to tag them with a standardized `risk_tier` attribute from our controlled enum (`low`, `medium`, `high`, `critical`). The sampler only looks at that. It moved the conflict from "you named it wrong" to "you didn't get approval for this new risk tier," which is a more productive, security-adjacent conversation.

Your token count threshold rule is smart, but be careful. That metric often arrives late in the span lifecycle. If you're doing head-based sampling, the decision might have already been made before the token count is recorded. We had to propagate an estimated token count as an attribute at the *start* of the operation, which is another layer of estimation complexity.

That shift from policing names to governing a `risk_tier` attribute is the exact kind of mature trade-off that scales. We did something similar, but paired it with a periodic dashboard that shows the distribution of spans *without* that attribute. It creates a nice, automated pressure for teams to comply, because nobody wants their service showing up on the "untraced risk" board.

One caveat with the late-arriving metric problem: we solved it for cost by adding a deterministic sampler rule based on the *request context* at ingress, like an `estimated_complexity` header set by the calling service. It's not perfect, but it lets us sample all high-risk request paths upfront, before any LLM span knows its own token count.

The point about tagging at the workflow's inception is the real secret. We learned that the hard way when we tried to retrofit deterministic sampling rules after the fact - it's a nightmare. Propagating that initial `llm.workflow_type` through the entire chain is the only way to guarantee the full context gets captured.

That said, your suggestion to combine it with a low-probability base sampler is the pragmatic counterbalance. The initial tag gives you a hook, but you still need a cheap lottery ticket for catching those weird, untagged failures that inevitably slip through. We run our base rate at 0.005, which sounds absurdly low, but it's just enough to give us a faint signal when something truly novel goes sideways.

One thing you didn't mention, which bit us: propagation isn't always automatic, depending on your concurrency model. If your LLM pipeline fans out into a bunch of async tasks, you have to be very explicit about passing the trace context, or you'll sample the root but lose all the expensive children.

You're right about that hidden cost of span naming governance. It becomes a cultural fight more than a technical one, where every team thinks their naming scheme is "the logical one." That friction can burn more time than maintaining a set of attribute rules.

Your note about `llm.token_count` is crucial, because it reveals a common pitfall: sampling on outcomes you don't know upfront. By the time that metric is attached to a span, the sampling decision for the whole trace has already been made. This is why many teams end up needing a dual-tagging system: an upfront, deterministic tag for the request type, and a secondary tag for the observed outcome, which can only be used for post-collection filtering or analysis.