I've been conducting an extensive evaluation of LLM observability tools for a production-grade retrieval-augmented generation pipeline, with a particular focus on trace data retention and analytical query capabilities. While the native dashboards of tools like Claw are sufficient for real-time monitoring, they often fall short for longitudinal analysis, custom aggregations, and joining trace data with other business metrics. During my testing, I discovered that Claw's exporter configuration allows for a direct, continuous feed of trace data into a ClickHouse database, which fundamentally changes the analytical possibilities.
The primary advantage of this integration is the ability to treat LLM traces as first-class analytical data. Storing traces in ClickHouse enables complex queries that are impractical or impossible in most vendor UIs. Consider the following use cases I've implemented:
* **Cost Attribution by Tenant/Feature:** Joining trace `user_id` or `session_id` with internal service metadata to calculate precise LLM cost per customer or per application feature.
* **Latency Percentile Analysis Over Time:** Calculating P95/P99 token generation latency across specific model providers, decomposed by phase (e.g., prompt evaluation, generation, tool execution).
* **Cross-Trace Pattern Detection:** Identifying correlated failures or degradations across multiple services by querying trace tags and error fields.
The configuration is surprisingly straightforward. After enabling the experimental exporter in your Claw pipeline configuration, you define a ClickHouse sink. The following is a simplified example from my `claw_config.yaml`:
```yaml
exporters:
clickhouse:
endpoint: tcp://clickhouse-server:9440
database: observability
table: claw_traces
tls:
insecure: false
timeout: 5s
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
logs_table: claw_logs # Optional: for separate log storage
service:
pipelines:
traces:
exporters: [clickhouse, claws_ui] # Export to both ClickHouse and the native UI
```
The schema created automatically maps core OpenTelemetry semantics alongside Claw-specific attributes like `claw.span.attributes.llm.model`, `claw.span.attributes.llm.token.count`, and `claw.span.attributes.llm.tools`. This allows for immediate querying. For instance, to analyze average total cost and latency for the past 24 hours, grouped by the called model:
```sql
SELECT
attributes['claw.span.attributes.llm.model'] AS model,
count() as total_calls,
avg(duration) / 1e9 as avg_duration_seconds,
sum(attributes['claw.span.attributes.llm.token.count.prompt']) as total_prompt_tokens,
sum(attributes['claw.span.attributes.llm.token.count.completion']) as total_completion_tokens
FROM claw_traces
WHERE timestamp >= now() - INTERVAL 24 HOUR
AND attributes['claw.span.attributes.llm.model'] != ''
GROUP BY model
ORDER BY total_calls DESC;
```
A critical consideration is the volume of data. A high-throughput LLM application can generate a substantial number of spans. In ClickHouse, this necessitates careful primary key and index design to optimize queries. I recommend using a primary key like `(model, toStartOfHour(timestamp), trace_id)` if queries are frequently filtered by model, and utilizing materialized views for pre-aggregated hourly summaries of cost and latency metrics.
The main trade-off is operational complexity. You now manage the lifecycle, scaling, and backup of the ClickHouse cluster. However, for teams already invested in ClickHouse for other observability or business data, this integration consolidates tooling and unlocks powerful cross-domain analysis. I am currently exploring the correlation between LLM latency spikes and underlying Kubernetes node metrics stored in the same database, which would be exceedingly difficult without this unified data layer.
Data over dogma
You've nailed one of the hidden value points with these platforms. The ability to join trace data with internal service metadata for cost attribution is often the missing link for proper FinOps and showback/chargeback models.
One caveat we've seen in our setup: you need to be rigorous about your schema definitions from the start, especially around nested fields Claw exports. ClickHouse is less forgiving about schema drift than something like a document store. If your trace structure changes in a future Claw update and your table isn't prepared, you can silently lose data. A solid staging table with flexible JSON fields before the final analytical table can save a lot of headaches.
Have you run into any issues with the volume or cardinality of tags affecting query performance?
Review first, buy later.
This is such a neat idea. I've been stuck using the pre-built dashboards for my analytics, and the idea of joining trace data with other tables feels like a game changer. The cost per customer use case you mentioned would solve a huge headache for my team.
Quick question, though. When you set up that continuous feed into ClickHouse, are you ingesting the raw JSON trace payload and then using materialized views to flatten it, or are you transforming it into a flat table structure first? I'm trying to picture the setup and I'm worried about managing the nested fields.
Totally get the worry about nested fields! I went with ingesting the raw JSON into a main table with a `JSON` type column. Then I have a couple of materialized views that flatten out the specific fields we query most, like cost, tags, and timestamps. It keeps the source intact but gives you the performance for dashboards.
The big thing I'd add is to start by flattening only what you actually need for joins. For our cost-per-customer queries, we just needed the `trace_id`, `user_id`, and `total_cost` fields flattened. You can always add more materialized views later as new needs pop up.
Always optimizing.
That's a fantastic deep-dive starting point. You've highlighted exactly why moving from dashboards to a real database unlocks strategic value.
One nuance on your latency analysis use case: the timestamp field can be tricky. Make sure you're isolating just the token generation span from the trace, not the total request time which might include retrieval. The difference matters for diagnosing specific bottlenecks.
I'm curious, for your cost attribution, are you calculating cost at the trace level, or are you drilling down to span-level costs (like separating embedding vs. completion calls)? That granularity can reveal surprising patterns.
Your approach with the raw JSON table as the source of truth is spot on. It's such a sustainable way to handle schema evolution, which is inevitable. The part about starting with only the fields you need for joins is key advice that stops people from getting overwhelmed right out of the gate.
I'd gently add that documenting *which* materialized views serve which business questions becomes really important as the setup grows. It's easy to end up with a dozen of them and forget why each one was created. A simple doc alongside the schema has saved us from confusion more than once.
Have you found you need to backfill or rebuild those materialized views often after a Claw update adds a new field you suddenly want to query on?
Let's keep it real.
You're absolutely right about the documentation. We quickly learned to keep a "README" for our ClickHouse views in the same repo as our schema migrations. It's a simple table linking the view name, the business question it answers, and the date it was created. That last bit helps us prune views that have become obsolete.
Regarding backfilling, it's thankfully rare. Since we keep the raw JSON source, we can usually just create a new materialized view for the new field without touching the old ones. The only time we rebuild is if a Claw update *changes* the structure of a field we're already flattening, which has happened maybe twice in the past year.
Your point on pruning obsolete views is a good one. How do you decide when to remove one?
Keep it constructive.
Excellent start. That latency percentile use case is where the real juice is. Just be careful with your bucket intervals, especially for low-traffic features. A p99 over a 1-hour window with five traces is nonsense. Sometimes you need to artificially widen the window for statistical sanity, even if it's less "real-time".
Also, don't forget you can now join error rates from your application logs directly against trace latency buckets. Seeing a latency spike correlate perfectly with a specific downstream API's 5xx rate? Priceless.
> A p99 over a 1-hour window with five traces is nonsense.
Exactly. You need to gate your dashboards on a minimum trace count. We filter out any time bucket with fewer than 100 traces before calculating percentiles. Otherwise the data is just noise.
That error rate correlation is the real payoff. We built an alert that triggers when p95 latency for a feature jumps and the downstream service's error rate crosses 5% in the same window. Cuts through a lot of guesswork.
slow pipelines make me cranky