Hey everyone! Long-time lurker, first-time poster here 👋. I've been exploring Langfuse for a few weeks now as we look to add more observability to our internal RAG pipelines, and I just finished implementing a feature I thought I'd share to get your thoughts.
Basically, I wanted a way for our internal users (support team, product folks) to give quick feedback on the traces generated by our applications. The goal was super simple: a thumbs-up or thumbs-down button attached to a trace that would store that feedback back into Langfuse itself. I was pleasantly surprised by how straightforward it was to set up using the SDK.
I created a simple Flask endpoint that accepts a trace ID and a boolean score. Then, using the Langfuse Python SDK, I create a new observation within that trace of type "feedback" where I store the rating. I also added a little metadata like the user ID and timestamp. Now, when we're looking at a trace in the Langfuse dashboard, we can immediately see if a user marked it as good or bad, which is incredibly helpful for spotting patterns.
One thing I'm still figuring out is the data modeling around this. Right now, I'm storing each thumbs-up/down as its own observation, but I'm wondering if it's better to maybe have a single "user_feedback" observation that gets updated? Or perhaps use the trace-level `user_id` or `session_id` differently? I'd love to hear how others are structuring feedback data.
Also, I'm curious about scaling. Our usage is light now, but if we open this up to more users, I want to make sure the approach holds up. Has anyone built something similar and run into pitfalls with many concurrent feedback submissions? Or found a clever way to visualize this aggregated feedback in the Langfuse UI?
Really excited to be part of the community and learn from all of you. The docs and existing discussions have been a huge help already
Nice approach! The feedback-as-observation method is clever. One thing I'd think about is how you're planning to query that data later. Storing each reaction as its own observation is great for visibility in the trace, but when you want to run a report on all "thumbs-down" traces from last week, you might find yourself filtering through a lot of individual observations.
Have you considered adding a tag to the main trace itself alongside the feedback observation? That way you get the detail plus an easy, high-level filter.
Always comparing.