I've been trying to implement a more structured prompt management system for our team's LLM features, and I keep seeing references to LangSmith datasets as a way to version prompts. The documentation is great, but I'm struggling with a practical workflow.
Specifically, I want to understand how you all are using datasets to track changes to prompt templates over time. For example, if we tweak the system prompt for our summarization feature, how do you systematically compare the outputs from the old prompt versus the new one against the same set of representative inputs?
My current approach feels clunky: I export a CSV of sample queries, save the old prompt version in a doc, then run a new script. I suspect I'm underutilizing the dataset features. How are you linking a specific prompt template version to a dataset run, and then analyzing the results?
I'm particularly interested in the comparison view. What constitutes a meaningful "test case" to add to a dataset for this kind of prompt A/B testing? Are you using the dataset primarily for regression testing (ensuring a change doesn't break known good cases) or for iterative improvement against a fixed benchmark?