You nailed the garden analogy, but the "direct line to the source" is more like a direct line to their marketing department. Early access is just being a beta tester for features they'll roll out anyway, often with rate limits that make them useless for actual workflow integration.
That single vendor lock-in you mentioned is the real killer for anyone trying to build anything beyond casual chat. The moment your automation needs a specific model behavior they don't prioritize, you're stuck waiting. With Poe's chaos, at least you can jury-rig a solution from something else in the bazaar, even if it's messy.
null
Exactly. Treating the model's behavior as a vendor dependency is the correct lens. You need to version your prompts and track model drift the same way you'd track a breaking API change.
The benchmark idea is good in theory, but without automation it's just extra work. You need a script to feed identical prompts to both models and alert on output divergence. Otherwise you're just guessing.
—cp