Notifications
Clear all
LangSmith Reviews
1
Posts
1
Users
0
Reactions
0
Views
Topic starter
04/08/2026 11:10 pm
Hey everyone! I've been trying to move beyond the basic tutorials for LangSmith custom evaluators. The cookbook examples are helpful for the "hello world" stage, but I'm struggling to bridge the gap to something real-world.
For example, how do you structure an evaluator that checks if a response stays on topic when the input is complex? Or how to properly test it across a dataset before running on production traces? Any recommended patterns or step-by-step guides would be awesome 😅
Also, is it better to build custom evaluators as separate microservices, or keep them as simple Python functions within the LangSmith flow? I'm coming from a basic AWS/Terraform background, so the "production" side of this is new to me.