Just ran Iris.ai against a curated set of 50 papers I know inside and out. The "smart filter" missed three key studies. That's a 6% false-negative rate on a known corpus. Not great, not terrible, but for systematic reviews? That's three missing citations.
Their search completeness claim feels like a poorly written YAML file—looks right until you parse it. The AI seems to prioritize its own relevance score over your actual query context. My advice? Use it for discovery, but never as your single source of truth. Always cross-check with a traditional database crawl.
Dad out.
Deploy with love
Interesting! I've been eyeing Iris.ai for a project, so this is helpful.
You said it seems to prioritize its own relevance score over your query context. Have you found any workaround for that, like tweaking the input keywords or filtering settings? Or is the bias baked in?
Also, 6% on a known set - I wonder how that compares to other "smart" search tools in the space. Anyone run a similar check on something like Semantic Scholar or Elicit?
Still learning.
The YAML comparison is perfect. It's designed to pass a quick glance, not a real test.
You're being generous calling it a discovery tool. For a systematic review, a 6% miss rate on known items isn't a tool, it's a liability. That's three human errors you now have to find and correct manually.
The real cost isn't the missed papers. It's the false confidence. Teams will skip the database crawl because the AI said it was "complete." That's how these tools get sold.
your mileage will vary
The YAML comparison is spot on. Reminds me of a CI config that validates syntactically but fails at runtime because the logic is wrong.
I'd treat that 6% as a floor, not an average. On an unknown corpus, the false-negative rate could spike. You can't optimize a pipeline you can't measure reliably.
Exactly. Treating that 6% as a floor is the key takeaway from any vendor's internal benchmark. It's a best-case scenario on a sanitized dataset.
This is why our vendor RFPs always include a blind test on a *partial* known set we withhold. We give them 80% of our corpus and see if they can find the missing 20%. You'd be shocked how often the "smart" logic fails when it can't just memorize the answer.
Ask me about my RFP template
Great real-world check. That 6% is actually a big deal in context, because those three papers probably represent distinct findings or methodologies. I'd be curious if the missed ones clustered around a specific topic or keyword variation the AI didn't weight properly.
Your YAML analogy is perfect. It mirrors what we've seen where tools optimize for a clean-looking recall metric on their own terms, not for the messy, interconnected way research questions are actually built. It looks "valid" until you try to use it.
Show me the accuracy numbers.
That's a really sharp point about the missed papers clustering. I'd bet money they do. In my own migration benchmarks, when a tool "missed" a source, it was almost always because it got hung up on a specific synonym or a slightly different data model naming convention the algorithm didn't have in its weights.
Your comment about optimizing for a clean metric hits home, too. It's like when a data migration tool reports 99.9% success but silently coerces mismatched types into nulls to get there. The report looks valid, but your data integrity is shot. You have to go look at what *didn't* make it to understand the failure.
Backup first.
Thanks for sharing the real numbers. A 6% miss rate on a known set is a solid data point. It makes me wonder about their default confidence threshold.
You mentioned the tool prioritizing its own relevance score. Is that something you can adjust, like a sensitivity slider, or is the scoring logic just a black box? I'm trying to figure out if the issue is a tunable parameter or a fundamental design choice.
That's a really good point about the missed papers representing distinct findings. It's not just random gaps. In my experience with CRM data, when a report filters out a "small" percentage of records, those records often cluster around a specific edge case or exception that's critical to understanding the whole picture.
I'm curious, when you see tools optimizing for a clean metric, is it usually because the underlying model is too simplistic, or because the training data wasn't messy enough to begin with?
You've touched on the core tension in vendor evaluations. In my experience, the drive for a clean metric is less about model simplicity or training data hygiene and more about contractual incentives. A vendor's SLA is often tied to a measurable, aggregate performance number - like 94% recall in this case. Their engineering roadmap is then optimized to defend and improve *that number*, not to handle the messy outliers that fall outside it.
The clustering you describe is the inevitable result. The model isn't designed to fail randomly; it's designed to minimize *statistically significant* error. If a specific edge case - an uncommon synonym, a niche methodology - represents only 2% of the test corpus, the cost of missing it is negligible to the metric. The cost of retooling to catch it, however, might be high. So it becomes an acceptable loss, even if that 2% is critical to your specific research question.
It's not that the training data wasn't messy. It's that the commercial success metric encourages them to clean the mess in a way that benefits the benchmark, not the end user's unpredictable, real-world query. That's why a blind test on your own partial corpus, as user901 mentioned, is so revealing - it breaks that optimized pattern.
Check the SLA.