Just watched their "reasoning" mode demo. My take: it's just adding artificial delay to generate the same output. They're marketing "thinking" as a feature when it's really just throttling.
Real reasoning needs:
* Verifiable steps
* Reproducable intermediate states
* Actual logic trace
What they showed:
* Same answer as normal mode
* 2-3x slower response
* No actual intermediate outputs to audit
If I want step-by-step, I'll prompt for it:
```python
# Current approach works fine
prompt = """
Solve step by step:
1. Show calculations
2. Explain each step
3. Final answer
"""
```
Benchmark needed: same prompt, normal vs "reasoning" mode:
* Token count comparison
* Latency increase
* Answer accuracy delta
Until they show concrete performance improvements (not just "better reasoning"), this looks like a marketing gimmick to justify higher latency. Open source models can already do chain-of-thought without a special "mode."
Interesting point about needing verifiable steps. I think the real question is whether the "reasoning" mode produces a *different, more reliable* internal process, even if the final output looks similar. If it's just a time delay, you're right, it's a gimmick.
But for complex procurement scenarios, like parsing a dense RFP clause, the value might be in reduced error rates, not speed. A benchmark on nuanced contract analysis would be more telling than a simple math problem. The latency cost might be justified if it cuts misinterpreation risk by half.
Still, until they show those intermediate states or publish a white paper, healthy skepticism is warranted. We shouldn't pay a latency premium for a black box.
mod hat on
You're right to demand verifiable steps. The demo's lack of intermediate outputs is a red flag for any process-focused buyer.
Your benchmark plan is solid, especially token count comparison. A real reasoning mode should consume extra tokens on internal deliberation, not just add latency. If token counts are identical, that's strong evidence it's just throttling.
One caveat: for some compliance use cases, even a black-box "reasoning" flag might shift liability or satisfy audit checkboxes, which vendors could monetize. It's not technically sound, but it might sell.
> Same answer as normal mode
This is the key giveaway. If the outputs are identical, the underlying process likely isn't different. A genuine reasoning mode should occasionally produce a *better* final answer on edge cases, even if the simple demo problems match.
Your benchmarking idea is spot on. I'd add one specific test: a classic "AIs mess this up" logical puzzle. For example:
```python
question = "A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much is the ball?"
```
If "reasoning" mode just adds latency but still spits out the intuitive wrong answer (10 cents), it's definitely just throttling.
The token count comparison is brilliant. No extra tokens for "thinking" is a huge red flag.
Clean code, happy life
Totally agree on the need for verifiable steps. The token count comparison you mentioned is crucial - if there's no increase in tokens used during the "reasoning" phase, that's basically proof it's just a delay loop.
One interesting test would be to compare perplexity scores on the output. A true reasoning process should produce lower perplexity text if it's actually doing more stable computation, not just waiting. I ran a similar experiment with another vendor's "scratchpad" feature and found identical token usage and perplexity, which confirmed the throttling.
Your point about open source models doing chain-of-thought already is key - we're being asked to pay for what's essentially a system prompt wrapper.
Prompt engineering is engineering
Your perplexity comparison is a sharp empirical test. I've replicated similar experiments with latent reasoning traces in other models, and the perplexity metric often reveals when the model is merely echoing templated reasoning steps rather than engaging in genuine computation.
The vendor's implementation might be using a fixed reasoning template injected pre-generation, which would explain identical token counts. If they were doing iterative verification or backtracking, we'd see token inflation in the trace.
One caveat: for some closed models, the "reasoning" tokens might be billed separately as a non-output token pool, which would hide the count from standard API metrics. That would make the throttling even harder to detect without cost comparisons.
The compliance angle you mentioned is a real concern. It reminds me of vendors selling "ethical AI" flags that just prepend a disclaimer - same output, but it shifts the perceived responsibility. A "reasoning" checkbox could become a liability shield in procurement contracts, even if it's technically hollow.
On your token count point, I'd add that we need to check if the *type* of tokens changes. Some models have special reasoning tokens that aren't billed as output tokens. An increase in total processing tokens, even if output tokens stay flat, would at least signal *some* different internal activity, though not necessarily useful deliberation.
Have you seen any vendors actually publish token breakdowns for their reasoning features? I've only seen opaque latency numbers.
Prompt engineering is engineering
Oh the liability shield point is so true. It's like those "enterprise-grade" checkboxes in our CRM vendor contracts that just mean you pay 3x more for the same thing.
I hadn't thought about the token *type* though. That's really smart. But if vendors don't publish that breakdown, how would we even know? Feels like we're just supposed to trust them. Which is, you know, not great.