Hi everyone. I’ve been reading a lot here about automated scoring and benchmark datasets, which is great, but I wanted to bring up a method I’ve been thinking about: simulated user interactions.
In my day job, I handle a lot of migrations (Google Workspace, CRM systems), and the real test of any tool is how it behaves under realistic, sometimes messy, user scenarios. I wonder if the same principle applies to evaluating LLMs.
Instead of just checking outputs against a static rubric, what if we built simulated users with specific goals and behavior patterns? For example:
* A "confused user" who asks follow-up questions with incomplete information.
* A "task-oriented user" trying to complete a specific workflow, like drafting an email from bullet points.
* An "escalating user" whose requests become progressively more complex within a session.
This could help measure things like:
* **Context retention** across a conversation.
* **Practical helpfulness** in achieving a user's goal, not just single-turn correctness.
* **Tolerance for ambiguity** and how the model guides the interaction.
I’m curious if anyone has tried building something like this, or if there are frameworks that already support this kind of dynamic evaluation. The static benchmarks are essential, but I feel we might miss how the model performs in the back-and-forth of a real use case.
Migration is never smooth.
That's a really good point. I see a lot of evaluation focus on that single-turn "answer," but in practice, a user's goal is almost always a multi-turn process.
The "escalating user" scenario you mentioned is spot on for testing practical usefulness. It reminds me of measuring a support agent's skill - you don't just grade their first canned response, you see how they handle the whole messy conversation to a resolution.
One caveat I'd think about is cost and scale. Running these simulated conversations is way more expensive than scoring static outputs. But for a final shortlist of vendors, or for a high-stakes use case like customer support, it could be invaluable. Have you looked at any of the tools that record actual user sessions for product analytics? I wonder if the methodology could be adapted.