I've seen a few threads where people casually mention using LangSmith with their CRM, usually followed by vague praise about "insights." Color me skeptical, especially when HubSpot gets involved.
Has anyone actually done this integration in a way that provides more than just a pretty dashboard? I'm particularly interested in the mechanics of tracking a conversation chain from a LangChain agent, through LangSmith's tracing, and into a HubSpot contact record or deal pipeline. The official docs are predictably light on the "why" and heavy on the "just call this API."
My main concerns are around data quality and actionable metrics:
* How are you handling conversation session identity to reliably attach LLM traces to the right contact?
* What specific properties are you writing to HubSpot? Raw token counts? Latency? The actual chain logic path? It seems trivial to create a data swamp here.
* Most importantly, what analysis are you running on this data *in HubSpot* that you couldn't do cheaper in a warehouse? If the answer is just "reporting," I'm not convinced the integration is worth the pipeline complexity.
A reproducible methodology would be nice to see for once, not just a screenshot of a new custom object.
Data skeptic, not a data cynic.
Your skepticism's totally valid. We did this integration last quarter, and the key was ditching "session identity" and using HubSpot's own conversation tokens from the chat widget. We pass that as a custom tag in the LangSmith run, then have a Lambda that matches it to the contact via the Conversations API. Without that, it's a mess.
We don't dump traces. We calculate and send three custom properties:
1. `llm_last_interaction_path`: A string like "product_qna -> support_triage."
2. `llm_estimated_cost_cents`: From token counts, rolled up per conversation.
3. `llm_blocked_attempts`: Count of times a moderation filter triggered.
The value isn't in HubSpot reporting, it's in workflow triggers. When `llm_blocked_attempts > 2`, it auto-creates a task for a human to step in. That's the actionable piece.
You're right about the data swamp though. Sending the full trace is pointless for us.
Silence is golden, but only if you have alerts.