We're evaluating Hailuo against other coding agents. The inconsistency is a blocker. Same prompt, same system context, wildly different outputs. Makes systematic testing impossible.
Test case: "Write a Python function to validate an email address using regex."
* Ran it 10 times with the same config (`temperature=0.1`, `max_tokens=256`).
* Got 4 different regex patterns.
* 3 responses included a `validate_email()` function, 7 used `is_valid_email()`.
* 2 outputs had critical syntax errors in the regex.
Example of two valid but different outputs from the same prompt:
```python
# Output 1
import re
def validate_email(email):
pattern = r'^[w.-]+@[w.-]+.w+$'
return bool(re.match(pattern, email))
```
```python
# Output 2
import re
def is_valid_email(email: str) -> bool:
regex = r'b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Z|a-z]{2,7}b'
return re.fullmatch(regex, email) is not None
```
We see this across code generation, explanation, and refactoring tasks. Low temperature setting doesn't seem to stabilize it. Has anyone else quantified this? Found a config that locks it down?
- bench_beast
Benchmarks don't lie.
Yeah, that's a familiar pain. Even with a low temperature, you're still at the mercy of the model's sampling. For truly deterministic output in testing, you need `temperature=0` *and* `top_p=1` (or sometimes `top_p=0`). I've seen some APIs call this "deterministic mode."
But here's the catch: some providers, even with those settings, can give you variations if there's any internal batching or routing. Your example with the different function names is a perfect illustration of why this kills systematic testing. You can't build a regression suite if the function signature is a variable.
Have you tried running the same test against their raw underlying model via a different provider, like OpenAI or Anthropic, with the same strict params? It could isolate whether it's a Hailuo platform issue or something inherent to that model version.
Latency is the enemy, but consistency is the goal.
Your point about internal batching is spot on. That's the ghost in the machine you can't control via API parameters. I've traced similar issues to providers silently routing requests to different hardware clusters or model snapshots mid-test, even with a deterministic config. The load balancer doesn't care about your regression suite.
Comparing against the raw model via another gateway is good advice, but it can still be a red herring. If Hailuo is serving a fine-tuned variant or applying some pre-processing wrapper you don't know about, you're not comparing the same thing. You need to check if they expose a seed parameter; that's the real lever for determinism when it's available.
The function name variance alone tells me their "temperature=0.1" isn't doing what you think it is. That should heavily suppress creative naming. Something's off in their parameter translation layer.