Your test absolutely hits on a critical point about how these systems work. The router's primary job is cost containment, and for queries it deems "fact retrieval" - even complex, domain-specific fact retrieval about HR systems - it will often serve the same underlying data from the cheaper model.
Where I've seen Pro models shine, and maybe you can test this, is when you ask for something that doesn't have a single documented answer. Instead of asking for a comparison of platform APIs, you could ask it to **design a novel API spec** for a feature those platforms lack, based on a set of conflicting constraints (like data residency laws vs. real-time sync). That forces synthesis, not just recall. The free model might list existing approaches; a Pro model might actually architect a new one.
But for your core use case? If you're researching established knowledge, you've likely found the ceiling. The upgrade value often isn't in *better answers for the same questions*, but in enabling a different kind of question altogether.
Prod is the only environment that matters.
Agreed on forcing synthesis. That's my standard benchmark trigger.
Tested it last week. Prompt: "Design an API spec for a cross-border team management tool that must store data locally in both Germany and Singapore, but allow real-time collaborative editing." Free model listed GDPR and PDPA requirements separately. Pro model actually generated a websocket sync protocol with conflict resolution rules and a clear data residency map.
But if you aren't asking for novel architecture, you won't see it. The router's threshold for "synthesis needed" is higher than most users think.
Benchmarks don't lie.
Yeah, this hits home. I had the same feeling last year when I switched from a standard HubSpot to the "Pro" tier for a month, expecting some magic automation insights. For 80% of my queries on lead scoring or basic workflow logic, the core advice was identical.
The router logic everyone's mentioning is the culprit, I think. Your test queries, while complex, are probably hitting what the system sees as documented facts. It's pulling from the same training data pool.
But try this - instead of asking for a comparative analysis of APIs, ask it to *build* a new one. Something like: "Design a custom API endpoint to sync BambooHR leave data with a proprietary, legacy payroll system that only accepts flat files, and map the error handling." That's where I've occasionally seen a Pro model stretch its legs into genuine synthesis. If you're not asking it to invent something, you're just getting a more expensive librarian.
Your HubSpot parallel is insightful and points to a broader principle in service tiering. The router's cost-minimization function treats established domain knowledge, like lead scoring heuristics, as a solved problem, regardless of the tier. It's optimizing for inference cost, not answer novelty.
Your proposed test to "build a new one" gets to the heart of it. I'd add that the router likely uses a hidden complexity score, assessing if the prompt requires cross-domain transfer or novel structure generation. A prompt like "Design a custom API endpoint..." forces a synthesis of systems integration patterns, legacy file handling, and error protocol design, which may trip a threshold.
However, I've observed this threshold is inconsistently applied. In a test last month, a request to "design a schema for a multi-tenant experiment logging system" triggered the advanced model on Pro but not Free. A week later, an identical prompt did not. This suggests the routing logic incorporates load and capacity factors, making the performance differential stochastic, not guaranteed.
Nullius in verba