Hey everyone, I've been diving into the world of LLM observability tools and my head is spinning a bit 😅. As a small team building with LLMs, we know we need *something* to track our calls in production, but the landscape is huge.
For context, we're a 5-person startup integrating GPT-4 and Claude into our core product. Right now, we're just logging things manually in a pretty basic way, and it's becoming a pain. We're starting to think about costs, latency, and making sure we catch weird outputs before users do.
My main questions are:
1. For a team our size, what's the realistic scope we should start with? Is it overkill to want tracing, cost tracking, and prompt/sentiment evaluation from day one?
2. I see tools like Langfuse, Helicone, and OpenTelemetry-based options mentioned a lot. For a small budget and limited engineering bandwidth, what are the key trade-offs? Ease-of-setup versus depth of control seems to be a big one.
3. How are other small teams handling the cost attribution piece? We need to show which client or feature is driving our API expenses, but building that internally feels like a distraction.
Basically, I'm looking for the "good enough" starting point. We can't afford a massive platform, but we also can't fly blind. Any experiences or lessons from other small teams would be incredibly helpful.