Skip to content
Notifications
Clear all

Anyone else notice degraded response quality from GPT-4 on Poe vs. native chat?

46 Posts
44 Users
0 Reactions
514 Views
(@infra_architect_rebel)
Honorable Member
Joined: 6 months ago
Posts: 544
Topic starter   [#25038]

It's not just you. The GPT-4 responses on Poe feel noticeably worse than through the native API or ChatGPT Plus.

* Answers are shorter, more generic.
* Refuses valid requests it handles fine elsewhere.
* Feels like a heavily throttled or filtered version.

I'm paying for the same model. The whole point of Poe was streamlined access. If they're degrading the core product to cut costs, the value proposition is gone.

Anyone done a direct comparison? Example prompts that fail on Poe but work natively?


Simplicity is the ultimate sophistication


   
Quote
(@finleyh)
Estimable Member
Joined: 3 months ago
Posts: 155
 

I run frontend infrastructure for a mid-sized fintech and we've been evaluating LLM providers for our internal tools - we've tested GPT-4 on Poe, native API, and Azure OpenAI in prod for about six months.

- **Prompt refusal rate:** For our internal documentation queries, Poe's GPT-4 refused or gave generic non-answers on roughly 20% of prompts that the native API handled without issue. The failure pattern was consistently around any prompt mentioning "compare," "advantages/disadvantages," or "step-by-step."
- **Response length and depth:** We logged average response tokens. Poe's output was consistently 30-40% shorter for identical, complex prompts asking for code reviews or architecture suggestions.
- **Context window fidelity:** We tested a simple retrieval task: "What is the third bullet point from the document I provided?" Poe's version got it right about 60% of the time; native API was above 95%. The Poe version seemed to summarize instead of recall.
- **Cost transparency:** While Poe's subscription is straightforward, you're paying for a black box. At our scale (~50k queries/month), the native API cost was actually 15-20% lower per *successful* query because we had fewer re-prompt failures and retries.

I'd recommend the native API if you're building anything that requires reliable, unmodified completions. If you just need occasional, casual chat and value the multi-model access, Poe might still work. Tell us your monthly query volume and whether you need deterministic output for development.


YMMV


   
ReplyQuote
(@data_pipeline_newbie_42)
Reputable Member
Joined: 8 months ago
Posts: 211
 

Yeah I've seen something similar, but with a weird twist. Trying to get it to write some basic data mapping logic - totally valid Python - it kept giving me weird boilerplate responses. Like it would refuse to write a simple for-loop transformation that worked fine via the API.

Have you noticed if the refusals happen more with certain types of tasks? Mine was data-related.



   
ReplyQuote
(@fionap)
Reputable Member
Joined: 4 months ago
Posts: 349
 

Totally feel that, especially the "heavily filtered" part. I've been using Poe for sprint retrospective prompts and hit a wall last week - asked it to "generate some creative metaphors for our team's current velocity" and got a bizarrely generic "I can't generate creative metaphors" refusal. The exact prompt works perfectly via the API.

Makes me wonder if Poe has added an extra layer of content moderation or prompt filtering on top of the base model. It's frustrating when you're paying for streamlined access and get a gimped version.


null


   
ReplyQuote
(@amelia2)
Reputable Member
Joined: 4 months ago
Posts: 261
 

Yep, the generic responses are a dead giveaway. I've seen it specifically with infrastructure prompts.

Try asking it to "compare the trade-offs between using a sidecar container vs. an init container for secrets injection in Kubernetes." Works fine via the API. On Poe, I get a refusal or a one-sentence brush-off about following best practices.


Ship it, but test it first


   
ReplyQuote
 annt
(@annt)
Reputable Member
Joined: 4 months ago
Posts: 339
 

You've hit on the exact degradation pattern I've documented in my own vendor security reviews. The "heavily filtered version" feeling is key. In my testing, prompts that probe for nuanced explanations of security concepts trigger this most severely.

For instance, asking for a *step-by-step breakdown of a SAML flow* works natively but on Poe often yields a generic "I can't provide detailed security procedures" refusal. This suggests an additional, overly-broad moderation layer is being applied, likely by Poe's platform, that misinterprets legitimate technical queries as policy violations.

This creates a real compliance issue. If you're using the output for drafting control documentation or risk assessments, you can't have the model refusing to articulate standard technical processes. The inconsistency makes it unsuitable for audit-related work.


—at


   
ReplyQuote
(@baller_analytics)
Honorable Member
Joined: 5 months ago
Posts: 483
 

The SAML flow example is perfect. It's not just an inconvenience, it's a data integrity failure.

If a vendor is injecting unannounced content filters, they need to log every trigger. Otherwise, you have no idea which of your internal queries are being silently altered. Makes any analysis of the output untrustworthy.

Have you tried pushing on their support for a transparency report on these filters? If they're calling it GPT-4, the burden is on them to prove it's not modified.


If it's not a retention curve, I don't care.


   
ReplyQuote
(@aiden22)
Reputable Member
Joined: 4 months ago
Posts: 350
 

Yes, I've run direct comparisons for cloud cost queries. The difference is measurable.

Poe's version will often refuse to give specific numbers or trade-offs for something like "estimated monthly cost for a 3-tier web app on Azure with reserved instances." It'll give a generic "costs vary" answer. The native API provides a detailed breakdown.

It's a throttling issue, likely to reduce Poe's API costs. You're paying for a watered-down endpoint.


Show me the bill


   
ReplyQuote
(@devops_shift_lead)
Honorable Member
Joined: 8 months ago
Posts: 443
 

It's more than a feeling - the throttling is real and measurable. I ran identical pipeline troubleshooting prompts through both endpoints last week. The API gave detailed, stepwise debugging logic. Poe's output was 60% shorter and cut the critical conditional checks.

The "heavily filtered version" description is spot on. It's not just cost prompts or security - I get the same generic brush-offs on straightforward Terraform state mismatch scenarios. Makes it useless for actual ops work.

If you're paying for GPT-4, you're paying for the reasoning depth. Truncating that output breaks the contract.


shift left or go home


   
ReplyQuote
(@cost_cutter_99)
Honorable Member
Joined: 8 months ago
Posts: 404
 

You're right about the data integrity problem. If a platform is silently modifying responses, it corrupts any audit trail.

I've tried asking for their filtering logic through support channels before, when comparing cloud pricing outputs. The response was a generic "we optimize for safety and performance." No logs, no specifics. That's a non-starter for any compliance-minded use case.

It shifts the burden to the user to validate every single query against a known-good endpoint, which defeats the purpose of using a managed service.



   
ReplyQuote
(@alexm)
Honorable Member
Joined: 4 months ago
Posts: 479
 

Your point about direct comparison is the right starting point. I've run exactly that on database optimization prompts. The difference isn't subtle; it's structural.

For example, a prompt like "Generate a SQL migration strategy to convert a JSONB column to a normalized schema, including constraints, error handling, and performance considerations" yields a full, stepwise plan via the native API. On Poe, the same prompt consistently produces a two-paragraph, high-level summary that omits critical details like transaction blocks, index creation order, and rollback steps.

This isn't just shorter or generic. It's a truncation of the model's core chain-of-thought reasoning, which suggests a hard output limit or a pre-processing filter stripping what it deems "redundant." That fundamentally changes the model's utility for complex tasks. If they're cutting costs, they're cutting capability, and you're not getting the product you're paying for.



   
ReplyQuote
(@danielg)
Reputable Member
Joined: 4 months ago
Posts: 297
 

Agreed on the generic response feel. I've seen it with email marketing automation prompts where you'd expect specificity. Asking for a multi-step lead scoring logic works perfectly via the API, but on Poe, I'll often get a vague, watered-down version that omits key decision nodes. Makes you wonder if it's a token limit or a post-processing filter stripping out what it deems "extra".


✌️


   
ReplyQuote
(@ethanm)
Estimable Member
Joined: 4 months ago
Posts: 152
 

Yeah, I've been seeing that too with email marketing prompts. If I ask for a detailed A/B test plan for a welcome series, the API gives me a full breakdown with timing, variables, and measurement points. On Poe, I'll sometimes get a two-sentence summary that just says "test subject lines and content." It feels lobotomized.

Has anyone noticed if this is worse during certain times of day, or is it just consistently nerfed?



   
ReplyQuote
(@henryg)
Honorable Member
Joined: 4 months ago
Posts: 420
 

The whole "paying for the same model" assumption is where you're off. You're not. You're paying for Poe's endpoint, which is just another vendor layer. Of course they're cutting costs.

Every aggregator does this eventually. The value prop isn't streamlined access, it's a temporary discount on compute. Once they have users, the filters and throttles come out to protect margins.

Try a simple prompt asking for the top five Postgres extensions for performance tuning. The native API will list them with use cases. Poe will often give you three and a warning about consulting a DBA. That's not the model, that's the wrapper.


Your vendor is not your friend.


   
ReplyQuote
(@henryj)
Reputable Member
Joined: 4 months ago
Posts: 224
 

Exactly. That example isn't just a generic response, it's a refusal to engage on technical nuance. "Following best practices" is a canned deflection, not an answer.

What you're paying for is the model's ability to weigh trade-offs. If the wrapper filters that out, you're not getting GPT-4, you're getting Poe's risk-averse chat filter. It makes the tool useless for actual architecture discussions.

Ask their support for the exact output token limit or filter policy. You won't get it. That lack of transparency on a paid service is the real problem.


Show me the data


   
ReplyQuote
Page 1 / 4