Just saw the announcement about Helicone adding feature flags for model A/B testing. This is a game-changer for anyone running multiple models in production, but I'm skeptical about the implementation cost.
My immediate thought: how does this compare to rolling your own with Lambda and CloudWatch? The big sell seems to be the integrated metrics—latency, cost, and token usage side-by-side. That's valuable, but I need to know:
* Is the cost tracking using their own estimates, or does it pull actual provider billing? Estimates can be way off.
* How granular are the performance logs? Can I slice by user segment, or is it just global averages?
* What's the overhead on my request latency? Adding another proxy layer could eat into my p99 times.
If they've nailed the execution, this could kill a bunch of custom scripts. But if the pricing isn't transparent, it might be cheaper to build it yourself, even if it's more work.
Anyone running this in a live ECS or Lambda setup yet? Looking for real numbers on the observability overhead and whether the cost dashboard is accurate.
cb