I've been conducting a systematic literature review using Elicit over the past several weeks, and while the tool is generally impressive, I've encountered a recurring and problematic inconsistency. On multiple occasions, I have found that for a well-defined query, Elicit will inexplicably omit highly relevant papers from top-tier journals in its initial results. These aren't obscure publications; I'm talking about missing seminal works from venues like *Nature*, *Science*, *JAMA*, or flagship conference proceedings in computer science. I have to discover these omissions by cross-referencing with traditional databases or Google Scholar, which defeats part of the efficiency purpose.
My methodology is typically as follows:
* I construct a precise query using a combination of keywords and "must include" phrases.
* I often utilize the "Paper search" function with the "Query only" mode to avoid the summarization step initially.
* I apply the relevant filters (e.g., publication year, study type).
* I review the first 50-100 results.
The missed papers are not borderline cases. They are directly on-topic, highly cited, and from authoritative sources. This raises concerns about the completeness of any systematic review conducted solely through Elicit.
I've begun to hypothesize about the root causes, based on my experience with other API-driven research tools and data pipelines:
* **Source Index Limitations:** Elicit's underlying corpus (primarily Semantic Scholar) might not have complete or timely coverage of all journals, or its ingestion pipeline could have gaps or delays.
* **Query Interpretation & Ranking:** The AI's interpretation of my query, and its subsequent ranking algorithm, might be prioritizing certain patterns (e.g., recent papers, certain keywords in the title) over pure relevance, inadvertently burying key older works.
* **Metadata Inconsistency:** If the metadata for a paper (authors, journal name) is formatted inconsistently in the source, it could affect retrieval. For example, "N. Engl. J. Med." vs "The New England Journal of Medicine."
* **Abstract vs. Full-Text Analysis:** Perhaps Elicit's relevance scoring relies heavily on abstract text, and a paper with a broadly written abstract might be ranked lower than a less significant paper with my exact keywords in its abstract.
Has anyone else performed a rigorous audit of Elicit's result sets against known gold-standard papers? I'm particularly interested in:
* Your own anecdotes of missed key papers.
* Any patterns you've discerned (e.g., misses older papers >5 years, misses certain publishers).
* Workarounds you've developed. My current process is inefficient:
1. Run initial Elicit search.
2. Extract key papers and authors.
3. Use Elicit's "Cited by" and "Similar papers" features on *those* found papers to hopefully surface the missed ones.
4. Run a separate, parallel search on a traditional database as a sanity check.
This feels like a data pipeline or relevance-tuning issue rather than a simple bug. From an integration perspective, the reliability of the source data and the predictability of the query engine are critical for building a trustworthy workflow. I'm concerned about the potential for systematic bias in literature reviews if this behavior isn't well-understood and mitigated.
- Mike
- Mike
Elicit's search backend is likely Semantics Scholar's API. It's not a full-text index and can have serious coverage gaps, especially for newer or high-impact papers that get embargoed or aren't in its specific corpus.
Check your source selection in settings. You might be searching only "Peer-reviewed" from a subset of databases. The "relevant papers" ranking also seems to prioritize its own relevance score over pure citation count or journal prestige.
For a systematic review, you can't rely on a single tool. Use Elicit for discovery, but your protocol needs a defined, reproducible search across multiple databases to be valid. This is a known limitation.
Show me the bill
That's a good point about the Semantic Scholar corpus, I hadn't considered embargoed papers. Your suggestion about source selection settings is helpful, I'll definitely check that.
It makes me wonder, how does this compare to using something like Connected Papers or ResearchRabbit for the discovery phase? I've heard they use different underlying networks, but I'm not sure if their coverage gaps are similar or if one tends to miss different types of key papers.
You've described the problem well, but your methodology still treats Elicit as a database. It's a discovery engine with a specific, limited corpus.
> I construct a precise query
That's likely part of the issue. Semantic Scholar's search is a best-effort graph traversal, not a boolean index. Your precise keywords might not align with the embeddings it uses for its "relevance" ranking, which seems to heavily favor its own metric over journal authority. I've missed a known *PNAS* paper on a topic while it surfaced three lesser-known preprints. The corpus just has holes.
For a systematic review, your protocol needs to list Elicit as a single source with this caveat, not your primary search tool. The validation step via Google Scholar isn't a bug in your process, it's a required control.
Your fancy demo doesn't scale.
That's interesting. It makes me wonder, are you using the same filters each time, or does the omission happen across different settings? I've noticed if I restrict to "highly influential" or a specific date range, some obvious papers drop out, but they appear in a broader search. Maybe its influence ranking just doesn't match real-world prestige.
Exactly. It's pulling from Semantic Scholar's corpus, which is a known subset. You can check their coverage page to see what they include.
The ranking is the bigger issue. It's not just missing papers; it's burying relevant ones that don't match its internal relevance score. I've had a top-cited paper from a major conference appear on page three while less-cited, tangentially related ones were top.
Treating it like a traditional database search will always give you these gaps. It's a starting point.
YAML all the things.