Hey everyone! I've been setting up LangSmith evals for a QA bot and realized my initial eval dataset feels static. I want to incorporate real user feedback to keep improving it. I've read the docs on "Feedback" and the API, but I'm a bit lost on the practical workflow.
Specifically, I'm trying to figure out the step-by-step from collecting a thumbs up/down in my app to using that to refine my eval criteria. Do you add feedback directly to a trace as it happens, or batch it later? How do you connect that feedback data point back to the specific test case in your eval dataset?
If you've built this loop, what was your approach? I'm using a Python FastAPI backend and a simple React frontend. Any pitfalls to watch for, like feedback skewing results if some users are more vocal than others?
Great question. The "step-by-step" you're missing often hinges on tagging. Here's a flow that's worked for me:
In your FastAPI backend, when you log the trace for the user's QA session, include a custom metadata tag like `run_type: production_user`. When you record the user's thumbs up/down via your React frontend (immediately is fine, batched is okay too), attach that same feedback to the trace using its unique `run_id`. The critical link back to your eval dataset comes later: you query LangSmith for all production runs with that tag and the associated feedback, then use that data to create new, targeted test cases in your eval dataset. For instance, a pattern of thumbs-down on finance-related questions becomes a new test case checking for accuracy on finance topics.
A pitfall to watch isn't just vocal users, but ambiguous feedback. A "thumbs down" doesn't tell you *why*. Consider adding a minimal categorical follow-up (e.g., "Incorrect", "Unhelpful", "Confusing") if you can, as it makes refining your eval criteria much more actionable than a simple binary score.
Keep it constructive.