Skip to content
Notifications
Clear all

ELI5: Why does it sometimes give answers that aren't in the papers?

5 Posts
4 Users
0 Reactions
63 Views
(@coffeelover)
Honorable Member
Joined: 4 months ago
Posts: 397
Topic starter   [#23796]

Alright, let's cut through the marketing. Elicit's whole thing is "answers from research papers." So why does it sometimes just... make stuff up?

Because under the hood, it's still an LLM. It's trained to generate plausible-sounding text, not to perfectly recall and cite facts. When the underlying papers are vague, contradictory, or just don't contain a clear answer to your specific query, the model defaults to its training—it fills in the gaps. Sometimes that's a synthesis, sometimes it's straight-up hallucination.

You're not getting a database query. You're getting a probabilistic text completion that's *weighted* by paper contents. Big difference. The citations look convincing, but the connection between the claim and the source can be tenuous. Always, *always* check the actual PDF.


Just my two cents.


   
Quote
(@bobw)
Reputable Member
Joined: 4 months ago
Posts: 342
 

Yep, that's the core of it. It's that probabilistic nature. I think the really sneaky part happens when the system *retrieves* a vaguely relevant paper and the LLM then writes a perfectly coherent summary that sounds authoritative, but the actual claim is just a slight, plausible extrapolation from the text. You click the citation and the paper says something adjacent, but not *that*.

I see a similar thing happen in API documentation sometimes - a tool will use an OpenAPI spec as context, but then confidently generate example code with parameters that don't exist. The source material is *right there*, but the pattern-matching brain fills the gaps wrong. Makes you want a "ground truth" toggle that just shows you the raw, highlighted excerpts and nothing else!


null


   
ReplyQuote
(@coffeelover)
Honorable Member
Joined: 4 months ago
Posts: 397
Topic starter  

Exactly. The "ground truth toggle" would just shift the burden. Then you're manually verifying excerpts against the original pdf anyway, which defeats the tool's purpose. The real problem is presenting this as a research assistant when it's a creative writing engine with a bibliography feature.


Just my two cents.


   
ReplyQuote
(@helenb)
Estimable Member
Joined: 4 months ago
Posts: 128
 

That bit about "weighted by paper contents" is helpful. I guess I assumed the weighting was more absolute.

So when it gives me an answer about tax treatment for a specific service, I'm not getting a legal citation. I'm getting its best guess based on phrases it's seen in similar contexts. That would explain why the summaries feel right but the details are off.

Does this mean tools like this are only useful for broad concepts, not precise regulatory questions?



   
ReplyQuote
(@integration_ian)
Honorable Member
Joined: 7 months ago
Posts: 396
 

You nailed it. That "probabilistic text completion" is exactly the problem, and it mirrors what I see in integration workflows.

I've watched middleware platforms do the same thing. They'll ingest an API spec, but when you ask for a data mapping, they sometimes infer a field transformation that *seems* logical based on naming conventions but isn't actually in the documentation. It generates a plausible connection that isn't there.

So the core issue is presenting pattern-matching as fact-retrieval. It's a design flaw, not a bug.


Integration is not a project, it's a lifestyle.


   
ReplyQuote