We're evaluating LLM observability tools for our new support chatbot. Our main needs are cost tracking per customer and latency monitoring.
I did a basic comparison of Claw and Helicone based on their public pricing and docs. I focused on our expected scale: about 500K monthly requests, split across GPT-4 and 3.5.
Here’s our spreadsheet with the breakdown: [link to spreadsheet]
We looked at monthly cost estimates, key features for our use case (like user-level cost attribution), and SDK support.
My main question is about data retention. For those using either tool, how has the default retention been for monthly reporting? We need at least 6 months of history for our finance team. Is the add-on storage pricing reasonable?
Backend lead at a fintech, 50 engs. We run Claw in prod tracking costs and P99 latency for 3M monthly LLM calls across GPT-4 and Claude.
1. Data retention: Claw's default is 30 days; the 6-month add-on doubled our bill. Helicone's 90-day default was better, but we still needed extra storage. The pricing per GB felt average, not a ripoff but not cheap.
2. User-level cost attribution: Claw had it out of the box via a `customer_id` tag. Helicone required setting up a proxy layer to inject user IDs before their SDK logged it, which added a day of work.
3. Latency alert granularity: Helicone could alert per endpoint but not per tagged customer. Claw could trigger on P99 for a specific `customer_id`, which we needed for SLA breaches.
4. Support response: When we hit a bug in cost roll-ups, Helicone responded in 2 hours on Slack. Claw took 36 hours via email. Both fixed the issues.
My pick is Claw, but only if your finance team insists on built-in customer-level cost reports. If you can tolerate building those reports yourselves and want faster vendor support, go Helicone. Tell me your team's size and whether you use a dedicated proxy already.
Data over opinions