Had a regression in a complex chain. The bug only triggered with specific user input, which was a pain to reproduce manually. Used LangSmith's trace replay with modified inputs to isolate it quickly.
Here's the workflow:
1. Found a past trace where the bug occurred in the LangSmith UI.
2. Clicked "Debug" -> "Open as Playground".
3. Modified the specific input field that I suspected was causing the issue in the playground interface.
4. Ran the replay. The chain executed with the new inputs, and the bug reappeared immediately.
Key points:
* The replay uses the exact same model/agent configuration as the original trace.
* You can modify **any** input parameter, not just the main prompt.
* This is faster than writing a custom script to call your chain with test data.
Example: The bug was a malformed SQL query from a natural language request. Changed the user input in the playground from "sales last month" to "revenue last quarter" and re-ran the trace. The same parsing error occurred, confirming the issue was in the SQL generation step, not the specific date logic.
-dk
Trust but verify, then don't trust.
Clever trick, but aren't you just debugging the symptom? You confirmed the SQL generation step breaks with a similar input, but the replay uses the *exact same* model configuration. What if the bug was introduced by a model version update on the provider side? Your replay would lock in the old config and you'd miss it entirely.
This assumes your trace is the perfect replica of production, which vendor demos always do. In my experience, the "same configuration" often drifts - different rate limits, latent space variations on the model's side, a temperature setting that's actually stochastic. You've isolated it *within LangSmith's simulation*. That's not the same as isolating it in your live system.
It's faster than a custom script, sure. But is it reliable, or just convenient? I'd still want a script to hit the actual deployed endpoint with a battery of modified inputs. Otherwise you're just watching the bug replay in a controlled zoo, not finding it in the wild.
cg