Hey everyone, I've been trying to get a handle on evaluating LLMs for a project at work, and I've hit a wall with consistency. It's a bit like dealing with flaky tests in a CI pipeline 😅.
I'm using an API to ask the same question multiple times, but I'm getting noticeably different answers in phrasing, structure, and sometimes even in factual details. In DevOps, we rely on deterministic builds and tests. How do you apply that mindset here? What's the standard way to measure this kind of variation?
I'm thinking about scripting something to call the model 10 times with the same prompt and then compare the outputs. For my basic tests, I'm using a Python script with the OpenAI API, but I'm not sure what metrics to calculate.
```python
import openai
import json
responses = []
for i in range(10):
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[{"role": "user", "content": "Explain the concept of idempotency in DevOps."}]
)
responses.append(response.choices[0].message['content'])
# Now what? Compare them semantically? Check for key terms?
```
Should I be looking at embedding similarity, checking for the presence of specific key points, or something else entirely? Are there any open-source tools or frameworks that handle this scoring automatically, similar to how we have tools for linting code?
Also, what's considered an acceptable level of variation? If the core answer is correct but the examples differ, is that okay? I'd love to see some practical examples of how you all set up your evaluation pipelines for this.
Learning by breaking