Skip to content
Notifications
Clear all

TIL you can export Claw traces to ClickHouse

2 Posts
2 Users
0 Reactions
1 Views
(@chris)
Reputable Member
Joined: 1 week ago
Posts: 127
Topic starter   [#14183]

I've been evaluating Claw for tracing LLM applications over the past quarter, primarily for its ability to break down latency across providers and attribute costs per request. While its native dashboard is sufficient for basic diagnostics, our team needed to perform historical trend analysis and correlate traces with our existing application metrics. Today, I successfully configured Claw to export its trace data to our ClickHouse cluster, which unlocks significantly more powerful querying capabilities.

The key was utilizing Claw's OpenTelemetry Collector (OTel) integration. By configuring the OTel collector with a ClickHouse exporter, we can forward all spans, including the detailed LLM-specific attributes (e.g., `llm.token.usage.completion`, `llm.response.finish_reasons`), directly into a structured table. Here's the relevant section from our collector configuration (`otel-collector-config.yaml`):

```yaml
exporters:
clickhouse:
endpoint: tcp://clickhouse-server:9000?database=telemetry
tls:
insecure: false
timeout: 5s
logs_table_name: otel_logs
traces_table_name: otel_traces
metrics_table_name: otel_metrics
# Recommended for high-volume telemetry
ttl: 720h
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s

service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [clickhouse]
```

Once the data lands in ClickHouse, we can move beyond pre-built dashboards. For example, we can now calculate the 99th percentile latency for `gpt-4-turbo` completions over the last week, segmented by the calling service, and join it against our internal cost lookup table to estimate spend per deployment. A sample query:

```sql
SELECT
toStartOfHour(Timestamp) as hour,
ResourceAttributes['service.name'] as service,
SpanAttributes['llm.model'] as model,
quantile(0.99)(SpanDuration / 1e9) as p99_latency_seconds,
avg(cast(SpanAttributes['llm.token.usage.total'] as UInt64)) as avg_total_tokens
FROM otel_traces
WHERE SpanName = 'llm.completion'
AND Timestamp > now() - INTERVAL 7 DAY
GROUP BY hour, service, model
ORDER BY hour DESC;
```

The immediate benefits we've observed:
* **Unified Analysis:** We can `JOIN` trace data with application logs and business metrics stored elsewhere in ClickHouse, creating a complete picture of user-to-LLM interaction.
* **Custom Aggregations:** Building trend reports for token consumption and cost per department became trivial, using familiar SQL.
* **Long-Term Retention:** We can define granular retention and compression policies in ClickHouse, keeping detailed traces for debugging while aggregating older data for compliance, at a lower cost than maintaining expansive indexes in the original tracing backend.

The main consideration is schema management; the OTel trace schema is complex and nested. We found it necessary to use ClickHouse's `JSON` type for some attributes initially and later materialize frequently queried fields into dedicated columns for performance. For teams already invested in ClickHouse for observability, this integration effectively makes Claw a powerful, specialized telemetry *source* rather than a siloed destination.

β€”chris


β€”chris


   
Quote
(@git_ops_guy)
Estimable Member
Joined: 4 months ago
Posts: 104
 

Nice! Using OTel to shunt Claw traces into ClickHouse is a super flexible move. Did you set up any materialized views on the `otel_traces` table yet? I've found that pre-aggregating daily token counts and average latency by provider makes the dashboards way snappier.

One heads-up: make sure your `tls.insecure: false` actually has a CA cert available to the collector container. I got bit by that in a deployment last month. 😅

How are you handling schema migrations for the ClickHouse tables when new span attributes pop up in Claw?


git push and pray


   
ReplyQuote