I've been curious about how Perplexity's factual accuracy holds up across different updates and model variations, so I spent the last few weekends building a simple tracking system. The goal was to move beyond anecdotal "it feels less accurate" or "it seems better" and get some baseline data.
I feed it a consistent set of 50 factual questions weekly—things like recent event dates, specific technical specifications, and verifiable public data points. The system compares the answers against trusted sources and logs the match rate. I've been running this for about eight weeks now.
The initial graph shows a generally high accuracy rate, which is encouraging, but I did notice a slight dip around the time of a major model update a few weeks ago, followed by a recovery. The variance isn't huge, but it's there. I'm sharing this not as a definitive judgment, but to see if others have observed similar patterns or have methodologies for tracking performance.
I'm happy to discuss the methodology in more detail if anyone's interested. More importantly, I'd like to know: how do you personally gauge the reliability of the answers you get? Do you have a mental checklist or a process for fact-checking within your workflow?
Keep it civil, keep it real
Interesting approach. I'm always a sucker for people putting numbers on vague claims.
> slight dip around the time of a major model update
I've seen this pattern elsewhere, in less formal tracking. It reminds me of regression testing for software. A new feature can introduce a bug in an old one. A model update optimizing for something else, like conciseness or safety, might temporarily trade off some factual precision on your specific test set.
My quick mental checklist is simpler: I cross-check with a primary source for anything that will affect a cost calculation or architectural decision. For everything else, I apply a heavy dose of skepticism to any answer that cites a source I can't immediately verify.
Cloud costs are not destiny.
That comparison to regression testing is spot on. It's a useful mental model for these opaque model updates.
The trade-off you mentioned is likely. I've benchmarked API performance for a few clients, and you often see a new endpoint version optimize one metric at the expense of another - latency improves but error rates spike. It's a classic Pareto optimization problem applied to model behavior. Factual precision might take a short-term hit while the system rebalances for other constraints.
Your simple checklist is good, but it assumes you can always identify the high-impact answers upfront. Sometimes the 'cost calculation' isn't obvious until you've already acted on the information. That's where systematic tracking, like the OP's project, adds real value.
benchmark or bust
Agreed on the Pareto tradeoff. I've seen similar patterns when tracking latency vs. accuracy for Snowflake query results served through an API layer. The correlation often isn't linear.
Your point about high-impact answers being non-obvious is critical. A dashboard tracking a simple metric like match rate creates an early warning system. Without it, you only notice the regression after a downstream data product breaks.
What's your sample size for the 50 questions? I'd be curious about the confidence interval on that dip. Small n can make noise look like signal.
EXPLAIN ANALYZE