I've been conducting a systematic evaluation of several research search engines, with a specific focus on their ability to handle precise, controlled vocabulary queries as part of my team's literature review workflow. My benchmark involves comparing recall and precision against a known gold standard set of articles retrieved directly from PubMed using Medical Subject Headings (MeSH). Consensus, while impressive in its LLM-powered summarization, demonstrates a significant and recurring flaw in this area.
During my testing, I constructed queries using specific MeSH terms paired with subheadings. For example, a precise PubMed query like `"Neoplasms"[Mesh] AND "therapy"[Subheading] AND "drug therapy"[Subheading]` returns a highly targeted set. When I input semantically similar natural language queries into Consensus ("drug therapy for cancer"), the results are broader, less structured, and miss key, highly-cited papers that are explicitly tagged with those MeSH terms in PubMed. This isn't a minor discrepancy; it points to a fundamental gap in the underlying indexing or retrieval architecture.
My hypothesis, based on the observed output, is twofold:
1. **Primary Data Source Limitation:** Consensus may not be ingesting or prioritizing the MEDLINE/PubMED database with its associated metadata fields (MH, MN, etc.) as a primary source. Instead, it appears to be relying on a general scholarly corpus (perhaps via Semantic Scholar or similar) where controlled vocabulary tags are not a first-class citizen in the document representation.
2. **Query Translation Gap:** Even if the data is present, there's no evidence of an automatic query translation layer that maps natural language to the relevant controlled vocabulary. This is a non-trivial NLP problem, but it's essential for professional-grade biomedical search.
The practical impact for a researcher or clinician is substantial:
* **Lower Recall:** Missed seminal papers that are precisely tagged.
* **Increased Noise:** Results contain articles where the topic is incidental rather than central, as controlled vocabulary is designed to denote the main focus of an article.
* **Broader Systematic Review:** This flaw makes the platform unreliable for the initial scoping phase of a systematic review, where capturing the entire relevant literature is critical.
I would be very interested to understand the technical stack behind Consensus's data ingestion pipeline. From an observability standpoint, this is a classic data quality issue—if the `MeSH` tags aren't being extracted and indexed as a high-priority field, they cannot be effectively queried. A potential workaround for the Consensus team could be to implement a post-retrieval ranking boost for articles that contain matching controlled vocabulary, even if the query was natural language.
Has anyone else performed similar comparative load tests on search precision, or have insight into how Consensus's document retrieval is actually implemented? The summarization is valuable, but if the foundational retrieval is incomplete, it risks summarizing an incomplete set of evidence.
—hj
Latency is a liability