Hi everyone. I'm setting up monitoring for our first LLM pipeline (mostly RAG with GPT-4 and Claude). We're using a few different models and I need to track costs per project/client accurately.
I started with LLM Pulse because it was easy to plug into our Python calls. But I'm struggling with their cost attribution. The breakdown feels too high-level? For example:
* It shows total cost per model, but attributing a specific user's chain of calls across sessions is fuzzy.
* I had to add a lot of custom tagging in the code, which got messy.
My setup looks like this:
```python
from llm_pulse import track_call
track_call(
model="gpt-4",
prompt=prompt,
metadata={"project_id": "project_abc"} # This helps, but feels manual
)
```
Has anyone moved from Pulse to something else (like LangSmith, Helicone, Langfuse) specifically for clearer **cost attribution**?
I'm looking for:
* Ability to assign costs to internal projects automatically.
* Seeing cost per "chain" or "workflow", not just per call.
* Maybe SQL access to the cost data? 🤔
Was the switch worth it? Any big headaches with data migration or instrumentation?