I've been evaluating Poe for some potential automation workflows—think generating alert summaries or drafting post-mortem sections from logs. The pricing page lists costs per message, but it's not clear how that translates when you're dealing with variable query lengths or complex instructions across different models.
Has anyone done a systematic comparison? I'm particularly interested in:
- Cost per *useful* output token for coding/structured tasks (Claude-3 Opus vs GPT-4)
- Whether the cheaper models (like Haiku) end up being more expensive in practice because they need more verbose prompts or produce less accurate results requiring re-runs
- Any hidden multipliers for longer context windows
I tried a quick test myself, sending the same prompt to a few models and calculating rough cost per 100 words of output, but my sample size is tiny. The results were all over the place, especially when I asked for JSON.
If you've run benchmarks, what was your methodology? Did you account for retries? I'd love to see some numbers or a simple script others could replicate.
- away
Good question, and I hit the same wall with their docs. My approach was to standardize on a "unit of work" - I used a task of generating a specific Salesforce Apex class from a detailed spec. This helped because it forced structured output.
I found Opus was actually cheaper *per successful completion* for that task, even though its per-message cost is higher. Haiku would often miss a requirement, forcing a re-prompt or a fix, which added tokens. So you're right to look at cost per useful token, not just raw output.
For the context window, I didn't see a separate multiplier on Poe, but remember that longer inputs chew through your included input tokens faster, which shifts the cost model if you're feeding in huge logs.
Did your quick test use the same "success condition" each time? That's the only way my numbers stopped being all over the place.
Still looking for the perfect one
Exactly. The "cost per successful completion" metric is the only one that matters for business logic. I've seen teams burn credits on cheaper models by having to implement validation layers and retry loops, which adds engineering overhead on top of the token cost.
One caveat to your Apex test: that's a deterministic output task. For more creative or exploratory work, the cost of a "miss" is lower, and a cheaper model's speed might win on total project cost, even with more iterations.