It's not just you. The GPT-4 responses on Poe feel noticeably worse than through the native API or ChatGPT Plus.
* Answers are shorter, more generic.
* Refuses valid requests it handles fine elsewhere.
* Feels like a heavily throttled or filtered version.
I'm paying for the same model. The whole point of Poe was streamlined access. If they're degrading the core product to cut costs, the value proposition is gone.
Anyone done a direct comparison? Example prompts that fail on Poe but work natively?
Simplicity is the ultimate sophistication
I run frontend infrastructure for a mid-sized fintech and we've been evaluating LLM providers for our internal tools - we've tested GPT-4 on Poe, native API, and Azure OpenAI in prod for about six months.
- **Prompt refusal rate:** For our internal documentation queries, Poe's GPT-4 refused or gave generic non-answers on roughly 20% of prompts that the native API handled without issue. The failure pattern was consistently around any prompt mentioning "compare," "advantages/disadvantages," or "step-by-step."
- **Response length and depth:** We logged average response tokens. Poe's output was consistently 30-40% shorter for identical, complex prompts asking for code reviews or architecture suggestions.
- **Context window fidelity:** We tested a simple retrieval task: "What is the third bullet point from the document I provided?" Poe's version got it right about 60% of the time; native API was above 95%. The Poe version seemed to summarize instead of recall.
- **Cost transparency:** While Poe's subscription is straightforward, you're paying for a black box. At our scale (~50k queries/month), the native API cost was actually 15-20% lower per *successful* query because we had fewer re-prompt failures and retries.
I'd recommend the native API if you're building anything that requires reliable, unmodified completions. If you just need occasional, casual chat and value the multi-model access, Poe might still work. Tell us your monthly query volume and whether you need deterministic output for development.
YMMV