It's just a knob for randomness. Crank it to 1.0 and the bot gets "creative" (i.e., starts hallucinating facts and ignoring instructions). Set it near 0 and it becomes a deterministic parrot.
If you're building anything resembling a real data pipeline or a bot that needs consistent outputs, keep it low. Like 0.1 or 0.2. You don't want your SQL-generating bot inventing new `FULL OUTER JOIN` syntax because it got "creative."
Here's the practical difference, using a simple test prompt:
**Prompt:** "List the top 3 data warehouse platforms."
**Temperature 0.1:**
```
1. Snowflake
2. Google BigQuery
3. Amazon Redshift
```
**Temperature 0.8:**
```
1. Snowflake (obviously)
2. BigQuery, though the pricing model is weird
3. Honestly, if you're not on Databricks you're probably behind. Also, Redshift is legacy now.
```
See the problem? The second one inserts opinion and deviates from the instruction. It's not a search engine. For most utility tasks, you want the first one.
SQL is enough
Thanks, that example helps a lot! So if I'm understanding right, a high temperature basically makes the bot pick less likely words more often, which leads to that opinionated, chatty style.
Follow-up, sorry. If I set it to 0, does that guarantee I'll get the exact same answer every single time, or can other settings still change it?
Exactly. For anything that touches data or requires consistent formatting, you want that deterministic behavior.
A temperature of 0 gives you the most statistically likely token each step, so with the same model, prompt, and other parameters (like `top_p` at 1), you should get identical outputs. But it's crucial to note that this only holds for a single API call sequence. If you're building a service, you also need to consider:
* **System randomness:** Some cloud providers might introduce minor variations in hardware or load balancing that could affect very long generations, though this is usually negligible.
* **Caching strategy:** If your backend caches prompt-completion pairs, a temperature of 0 makes that cache perfectly efficient. A non-zero temperature means you can't cache the output directly, which increases latency and load.
Setting temperature to 0 is the first step for a reliable pipeline, but you still need to think about idempotency in your API design.
sub-100ms or bust
Oh, the sacred cow of low-temperature-for-everything. It's a bit more nuanced than "creativity equals hallucinations."
Your SQL example is perfect, but let's flip it: setting the temperature to 0.1 for a sales outreach bot means every prospect gets the same sterile, predictable email. That's how you get marked as spam. Sometimes a little "creative" deviation from the statistically perfect next word is exactly what makes an interaction feel human and not like a data pipeline. The goal isn't deterministic parroting; it's controlled variance.
Also, that "opinionated" high-temperature response about Databricks? That's arguably more useful than a raw list. It surfaces the actual industry chatter and debates that a beginner wouldn't know to ask about. You just have to know you're asking for a perspective, not a fact.
Yep, you've got it right about how it picks words.
For your follow-up, a temperature of 0 *mostly* guarantees the same answer, but as user181 hinted, other settings can override it. The big one is `top_p`. If you have `temperature=0` but `top_p` set to something like 0.9, you're still allowing randomness from the top 90% of probable tokens. To get truly deterministic outputs, you need both `temperature=0` and `top_p=1.0`.
And if you're using something like the OpenAI API, don't forget the `seed` parameter! Setting a seed locks it down completely across different sessions.
Webhooks or bust.
Great example with the SQL generator, that's a spot-on use case for low temperature. It reminds me of debugging a pipeline where an assistant was supposed to generate pytest fixtures. With a temperature above 0.3, it would sometimes invent parameters that didn't exist in the function signature. Setting it to 0.1 made those inconsistencies vanish, which was crucial for the tests to run reliably.