Hi everyone! I'm pretty new to using Poe for work, and I've been testing a few bots to help with research summaries.
I wanted to see which bots were most accurate and cost-effective, so I built a simple tracker in Airtable. It logs each query, the bot used, the answer quality (I score it 1-5), and the estimated cost from the Poe pricing page.
So far, it's been eye-opening! Some of the more expensive bots aren't always giving better answers for my tasks. Has anyone else tried tracking this stuff? I'd be curious how you measure "accuracy" or if there are better metrics to use. 😊
That's a smart approach! I do something similar for tracking email platform deliverability, but I've also started factoring in time-to-response for my sales engagement bots. A costlier bot might be "accurate" but if it takes three times longer to generate, it's not necessarily more effective for my workflow.
For accuracy, I sometimes add a second column for "completeness" - did it answer the full question or just part of it? That's helped me differentiate between a technically correct but superficial answer and a truly useful one.
What kinds of research summaries are you working on? The best metric might shift a bit depending on whether you need precise data extraction or more conceptual synthesis.
Less hype, more data.
That's really clever! I've been meaning to set up something like this. My team just uses whatever bot is "recommended" and we have no idea about cost or if it's even the best one.
How do you handle the "estimated cost" part? I get confused because Poe's page shows cost per message, but some of my queries are way longer than others. Do you just use the flat rate?
CloudNewbie