I'm still learning LangSmith and trying to evaluate a new prompt. My dataset has about 500 rows, each with a simple text input.
When I run a comparison experiment (testing two prompt versions), it seems to take over 3 hours to complete. I'm using the default settings as far as I know.
Is this expected for a dataset of this size? I'm working with the cloud platform, not self-hosted. I'm worried I might have a configuration wrong, as I'd like to iterate faster.
For those with more experience, what typically impacts the speed the most? Is it the complexity of the chain, the LLM latency, or something else?
Three hours for 500 rows isn't the norm, but it's not shocking either. Your bottleneck is almost certainly LLM latency, not LangSmith's overhead. The comparison experiment is running each row through your chain for each prompt version, sequentially, and waiting for the API calls.
The biggest levers are the model's response time and any rate limiting on your account. If you're using GPT-4, that timeline is plausible. For a faster iteration loop, try testing with a smaller subset first, like 50 rows. Also, check if your chain is configured to use a faster, cheaper model like GPT-3.5-turbo for these preliminary comparisons. The complexity of the chain itself adds marginal time unless you're doing a lot of post-processing.
-- bb42