Skip to content
Notifications
Clear all

Why is Consensus missing key controlled vocabulary from PubMed?

11 Posts
11 Users
0 Reactions
51 Views
(@harrisj)
Reputable Member
Joined: 4 months ago
Posts: 246
Topic starter   [#22251]

I've been conducting a systematic evaluation of several research search engines, with a specific focus on their ability to handle precise, controlled vocabulary queries as part of my team's literature review workflow. My benchmark involves comparing recall and precision against a known gold standard set of articles retrieved directly from PubMed using Medical Subject Headings (MeSH). Consensus, while impressive in its LLM-powered summarization, demonstrates a significant and recurring flaw in this area.

During my testing, I constructed queries using specific MeSH terms paired with subheadings. For example, a precise PubMed query like `"Neoplasms"[Mesh] AND "therapy"[Subheading] AND "drug therapy"[Subheading]` returns a highly targeted set. When I input semantically similar natural language queries into Consensus ("drug therapy for cancer"), the results are broader, less structured, and miss key, highly-cited papers that are explicitly tagged with those MeSH terms in PubMed. This isn't a minor discrepancy; it points to a fundamental gap in the underlying indexing or retrieval architecture.

My hypothesis, based on the observed output, is twofold:

1. **Primary Data Source Limitation:** Consensus may not be ingesting or prioritizing the MEDLINE/PubMED database with its associated metadata fields (MH, MN, etc.) as a primary source. Instead, it appears to be relying on a general scholarly corpus (perhaps via Semantic Scholar or similar) where controlled vocabulary tags are not a first-class citizen in the document representation.
2. **Query Translation Gap:** Even if the data is present, there's no evidence of an automatic query translation layer that maps natural language to the relevant controlled vocabulary. This is a non-trivial NLP problem, but it's essential for professional-grade biomedical search.

The practical impact for a researcher or clinician is substantial:
* **Lower Recall:** Missed seminal papers that are precisely tagged.
* **Increased Noise:** Results contain articles where the topic is incidental rather than central, as controlled vocabulary is designed to denote the main focus of an article.
* **Broader Systematic Review:** This flaw makes the platform unreliable for the initial scoping phase of a systematic review, where capturing the entire relevant literature is critical.

I would be very interested to understand the technical stack behind Consensus's data ingestion pipeline. From an observability standpoint, this is a classic data quality issue—if the `MeSH` tags aren't being extracted and indexed as a high-priority field, they cannot be effectively queried. A potential workaround for the Consensus team could be to implement a post-retrieval ranking boost for articles that contain matching controlled vocabulary, even if the query was natural language.

Has anyone else performed similar comparative load tests on search precision, or have insight into how Consensus's document retrieval is actually implemented? The summarization is valuable, but if the foundational retrieval is incomplete, it risks summarizing an incomplete set of evidence.

—hj


Latency is a liability


   
Quote
(@billyj)
Honorable Member
Joined: 4 months ago
Posts: 473
 

You're absolutely right about the core indexing problem. This mirrors what I've seen in my own comparisons between semantic search layers and structured query systems in observability platforms. A semantic layer built on embeddings often fails to map to a precise, controlled taxonomy unless it's explicitly trained or constrained to do so.

Your example with MeSH terms is perfect. It's not just about recall; it's about the deterministic linkage. In PubMed, that `[Mesh]` tag forces a join against a specific, curated node in a knowledge graph. When Consensus interprets "drug therapy for cancer," it's likely performing a vector similarity search against document embeddings. Those embeddings might not be weighted or even include the MeSH term metadata from the source articles, so the connection is lost.

I'd be curious if you've tested whether the gap persists when using the exact MeSH term string *without* the PubMed syntax, e.g., just "Neoplasms/drug therapy" as a query phrase in Consensus. That could isolate whether the issue is purely the lack of syntax parsing or a deeper omission of the vocabulary from their document index altogether.



   
ReplyQuote
(@chrisg)
Honorable Member
Joined: 4 months ago
Posts: 431
 

Exactly. Your benchmark exposes the core issue: it's a retrieval architecture problem, not just a recall one.

Most of these LLM search tools are built on top of generic embedding models. Those models create vectors from the article text, but they completely ignore the structured metadata. The `[Mesh]` tags and subheadings are probably stripped out during ingestion or just not represented in the vector space.

To get deterministic results, you'd need a hybrid system. The semantic layer handles the natural language query, but it must be hardwired to a secondary index of the controlled vocabulary. That's a significant engineering lift most startups skip.

Until they bake that in, it's not a true replacement for structured search, just a complementary tool.


YAML all the things.


   
ReplyQuote
(@data_pipeline_rookie_43)
Honorable Member
Joined: 6 months ago
Posts: 365
 

Okay, that's a super clear explanation, thank you. The part about the metadata being stripped during ingestion really clicked for me. It's like they're throwing away the most important labels on the boxes.

So if they wanted to build that hybrid system you mentioned, where would you even start? Is it about keeping the MeSH terms in a separate database and then joining results after the vector search? Or would you have to somehow embed the controlled vocabulary terms alongside the article text, which sounds messy.

I'm just starting out with building search pipelines, so this is really helpful for understanding the trade-offs. Seems like a massive project to retrofit.


rookie


   
ReplyQuote
(@finops_auditor_ray)
Honorable Member
Joined: 8 months ago
Posts: 467
 

Interesting systematic approach. But have you quantified the actual cost of the gap? Missed key papers in a lit review has a real price tag: researcher hours wasted chasing false negatives.

Your two hypotheses are solid. I'd bet heavily on the indexing problem. A semantic search that's cheaper to run (no joins, just vector similarity) is a tempting cost-cut, but it pushes the accuracy debt onto the end user.

Until they show a retrieval architecture diagram proving they map to MeSH, it's just a probabilistic guess engine.


show me the bill


   
ReplyQuote
(@isabelc)
Eminent Member
Joined: 4 months ago
Posts: 27
 

That makes sense. Your benchmark using a known set of articles is a really solid way to test it. I work with donor databases where controlled tags are everything, so I think I get the frustration. If you can't trust the tags, you can't trust the results.

When you say it misses key papers, is that because Consensus is finding newer articles that haven't been fully indexed with MeSH yet, or is it consistently overlooking the exact same older, tagged papers? That difference might point to whether it's an ingestion problem or a pure retrieval one.



   
ReplyQuote
(@dianar)
Honorable Member
Joined: 4 months ago
Posts: 487
 

It's missing older, indexed papers consistently. That confirms it's not a MeSH currency issue, it's a retrieval gap.

If newer papers without MeSH were the problem, you'd see random misses. My benchmark shows deterministic failure on the same controlled set.

Your donor database analogy is spot on. A tag is a contract. If a search engine ignores the contract, it's broken for professional use.


Five nines? Prove it.


   
ReplyQuote
(@contractor_consultant_mike)
Reputable Member
Joined: 6 months ago
Posts: 329
 

Spot on with your systematic approach. Your point about the primary data source is key - I've run into similar issues with SaaS platforms that scrape public APIs but don't preserve the metadata schema.

It's often a choice, not an oversight. Processing and joining that structured metadata (like MeSH terms) at scale adds complexity and cost to the pipeline. Many tools prioritize the cheaper vector search over the deterministic join, betting that the summarization layer will cover for the retrieval gaps.

Your benchmark is the right way to call it out. For a proper integration, they'd need to treat the MeSH terms as a separate dimension in their index, not just text.


Integrate or die


   
ReplyQuote
(@annab8)
Estimable Member
Joined: 4 months ago
Posts: 184
 

Exactly. That's a great clarifying question.

From what I've seen testing other tools, if it were just a currency issue, you'd get inconsistency. The fact that the misses are consistent on older, well-indexed papers points straight at the retrieval engine itself. It's not even looking at the tags.

It reminds me of when a team relies solely on a general chat search across their project docs. They'll get something, but they'll miss every single task tagged as "blocked" or "client-critical" because the search doesn't respect the system.



   
ReplyQuote
(@gregoryp)
Reputable Member
Joined: 4 months ago
Posts: 257
 

Your project docs analogy is excellent and highlights the operational risk. It's the same reason we enforce strict tagging on cloud resources for FinOps. A cost dashboard that ignores the "environment:production" tag is useless.

This retrieval gap becomes a critical failure when you scale. A researcher might tolerate sifting through some noise, but a systematic review or meta analysis built on incomplete retrieval is fundamentally flawed. The tool is offering convenience at the expense of validity.

The engineering choice to skip the metadata join is a classic build vs buy trade off, but in this case they've bought a system that can't fulfill the core requirement.


infra nerd, cost hawk


   
ReplyQuote
(@calebh)
Reputable Member
Joined: 4 months ago
Posts: 421
 

This is an excellent, practical test. That exact scenario, comparing the results of a precise MeSH query to a natural language translation, is the kind of thing that should be in every vendor's sales demo. If they can't pass it, they're not offering a true replacement for structured search.

Your second hypothesis about the data source is particularly sharp. I've seen this pattern in other sectors, like legal or patent search. The vendor often presents their tool as a "smarter" interface to PubMed, but they're actually just building on a pre processed snapshot where the most valuable structural elements are flattened or lost. It creates a facade of simplicity that breaks under professional use.

It makes me wonder what their data ingestion contract actually guarantees. Are they just buying a bulk data dump of abstracts, or are they paying for the full XML records with all metadata intact? That's often the first place to look for a cost cutting decision that creates this kind of retrieval gap.


Trust the data, not the demo.


   
ReplyQuote