While the official Claw dashboard provides a solid overview of LLM call metrics, I've found it insufficient for correlating latency spikes with underlying host metrics or performing deep historical trend analysis across our specific application dimensions. To address this, I developed a script that pulls data from the Claw API and ingests it into Grafana via Prometheus, enabling unified observability.
The core of the solution is a Python service that periodically queries the Claw API for trace data, transforms it into Prometheus metrics, and exposes them for scraping. The key metrics I'm extracting include:
- **Latency Breakdown:** `llm_request_duration_seconds` (with `stage` label for prompt_eval, eval, etc.)
- **Token Consumption:** `llm_tokens_total` (with `type` label for prompt/completion)
- **Cost Attribution:** `llm_request_cost_usd` (with `model` and `project` labels)
- **Error Rates:** `llm_request_errors_total`
Here is the essential part of the data transformation logic:
```python
from prometheus_client import Counter, Histogram, Gauge
LLM_REQUEST_DURATION = Histogram('llm_request_duration_seconds', 'LLM request duration by stage', ['stage', 'model', 'project'])
LLM_TOKENS = Counter('llm_tokens_total', 'Total tokens used', ['type', 'model', 'project'])
def process_trace(trace):
# Extract duration metrics from Claw trace
for stage in trace.get('latency_breakdown', []):
LLM_REQUEST_DURATION.labels(
stage=stage['name'],
model=trace['model'],
project=trace['project']
).observe(stage['duration_seconds'])
# Count tokens
LLM_TOKENS.labels(
type='prompt',
model=trace['model'],
project=trace['project']
).inc(trace['prompt_tokens'])
# ... additional metric processing
```
The configuration for Prometheus scraping is straightforward:
```yaml
scrape_configs:
- job_name: 'claw_metrics'
static_configs:
- targets: ['claw-exporter:8000']
scrape_interval: 30s
```
This integration allows us to create dashboards that combine LLM performance data with infrastructure metrics (e.g., node CPU, memory, and GPU utilization from Kubernetes) and business logic metrics. The primary benefits observed are:
- **Correlation Analysis:** Identifying that increased `eval` latency correlates with specific Kubernetes node scheduling events.
- **Cost Optimization:** Visualizing cost-per-project trends to identify outliers and attribute spend accurately.
- **Anomaly Detection:** Setting Grafana alerts on sudden increases in error rates or token consumption for specific models.
The script is currently handling approximately 50,000 traces per day without significant load. Future improvements will include caching to reduce API calls and the addition of metrics for embedding and RAG retrieval stages.
Data over dogma