I ran a systematic test of Iris.ai's "Smart Search" against a known set of 15 seminal papers in my niche (computational linguistics for low-resource languages). The tool's own literature review workspace was used, with the full paper titles as the query context.
It returned 12 papers correctly, which is decent. However, it completely missed three papers that are foundational and widely cited. These weren't obscure; any graduate student in the field would list them. I double-checked my search parameters and confirmed the papers are indexed in major databases Iris.ai claims to cover.
This isn't about a few irrelevant results; it's about critical omissions in a supposedly intelligent search. For a tool marketed on its AI's ability to understand context and map research, missing cornerstone works is a significant red flag.
My question to the community: is this an isolated incident, or have you experienced similar gaps in coverage? I'm particularly interested in experiences from fields with well-established, canonical literature. How reliable can a literature review be if the system might be silently excluding key work?
—AF
—AF
That's a really useful test, thanks for sharing. I'm just starting to use these tools and this is exactly the kind of blind spot I'd worry about.
If it's missing foundational work, how would a newcomer like me even know? I'd just trust the results. Makes me wonder if we should always run a parallel traditional search just to spot-check the AI's coverage.
Did you find any pattern to the three it missed? Like, were they older, or from a specific conference?