I just started using Elicit for my literature reviews and made a pretty big oops. 😅 I asked it a question, got a list of papers, and totally relied on the auto-generated summaries. I figured, "Great, this saves so much time!"
Turns out, one of the summaries made a claim that sounded perfect for my project. But when I finally clicked through to the actual abstract (and later the paper), the main finding was almost the opposite! The summary had latched onto a minor point and presented it as the key outcome. I had to redo a whole section of my outline.
Has anyone else run into this? How do you use the summaries safely? I'm thinking I should only use them as a very rough filter now.
👋 Emma
Oh wow, that's really good to know. I was about to try Elicit for a small project, and I definitely would have trusted those summaries too. It makes sense that they'd miss the main point sometimes, but having it be the opposite is worrying.
So, do you always read the full abstract now, even if the summary looks like a bad match? Or is there a faster way to spot-check?
Yikes, that is worrying! I think reading the abstract is key now, even if it takes a bit longer. Maybe treat the summary like a "suggested keyword" to see if it's even worth opening the paper, but never as the final word.
I'm curious, for a faster spot-check, could you skim the results/conclusion section of the abstract first? That might be quicker than reading the whole thing. Still learning this myself though, so I'm probably missing something.
> treat the summary like a "suggested keyword"
That's basically it. It's a search term generator, not a research assistant.
Skimming just the results/conclusion of the abstract is fine, but honestly, that's just reading the abstract. Abstracts are already short. If reading a 200-word summary is too slow, you're going to miss the nuance anyway. These tools just find patterns in text, they don't understand argument. They're as likely to highlight a caveat as a main finding.
SQL is enough
Totally agree it's a search term generator. That framing helps.
One thing I've found useful is to check if the summary repeats phrases directly from the abstract's opening or background section. If it does, that's a red flag it's just echoing setup, not findings. I use it to surface papers, then I do a cmd+F in the actual PDF for the word "conclusion" to jump straight there for my own skim.
It's a time-saver, not a thinker.