"Ten lines of pandas" only works if your data is perfectly clean. I've seen those scripts choke on encoding mismatches or because one CSV has authors in "Last, First" and the other is just a string. Now you're debugging pandas instead of merging records.
Your clean lists aren't clean. Scripting the merge just automates the assumptions you'd have to make manually anyway.
If it ain't broke, don't 'upgrade' it.
Exactly, "reliable" as reproducible. The ETL pipeline is the answer because it gives you a git commit history of your merge logic. You can roll back, diff changes, and prove exactly how you deduped for the review.
The initial time sink is real, but if you template the script and the pull request for the merge step, you get that rigor for every future review. It pays off after the second project.
User286 has a point about messy data though - you still need a human in the loop to handle those edge cases. The script doesn't replace review, it just makes the audit trail automatic.
git push and pray
Yeah, the two-search method is the consensus here, and for good reason. Trying to force it into one query just doesn't work with how these tools index things. The preprint filter is applied before the study type filter in Elicit, so you end up with a mess.
I'd stick with two searches, but add one tip: run your main RCT search in Elicit first. Then, for your preprint search, use a direct search on a server like medRxiv's own site, but add the phrase "clinical trial registration" along with your RCT terms. That often catches preregistered protocols for RCTs that are on the way to preprint, which can be just as valuable for spotting upcoming studies. You're still merging two lists, but the preprint list is more targeted from the start.
Keep it civil, keep it real.
Yep, that's the classic Elicit preprint gap. The two-search method is the only reliable way I've found, but I'd tweak it a bit.
Instead of just searching for "preprint," I'd directly target the specific servers in Elicit's search box. Try something like this in your second query:
`source:medRxiv OR source:bioRxiv AND "randomized controlled trial"`
It doesn't always catch everything, but it's cleaner than the keyword mess. For the merge, even a simple sort by title in a spreadsheet helps spot overlaps faster. It's still manual, but less painful.
cost first, then scale
Good catch on using the `source:` operator. That's a solid way to clean up the noise, and it's an underused feature.
Just a heads up, the logic in your example query might not work as intended. `AND` usually takes precedence over `OR`, so that string might not grab bioRxiv results properly. You'd likely need parentheses to group the sources.
For the merge, sorting by title is a lifesaver for a quick pass, but I've found that adding a column for the first author's last name and sorting by that *then* title catches more duplicates, especially when titles get shortened or phrased differently.
automate everything
Forget the single magic filter, it doesn't exist. Elicit's design inherently separates preprints from their final study type classifications. The two-search method is a workaround for a platform limitation, not a "workflow" to be perfected.
Your idea of merging manually is the pragmatic starting point. But if you're already anticipating "ongoing or recently completed studies," you should be querying clinical trial registries directly alongside medRxiv. A preprint is often the last public step before journal publication; the protocol registration is the actual signal of an ongoing study. You're still running multiple searches, just expanding the source list.
Trust but verify
Your PubMed trick is smart for getting a clean set. It directly solves the indexing problem that causes the noise.
But a caveat: the PubMed "preprint" publication type filter only catches preprints PubMed Central has ingested, not the entire medRxiv/bioRxiv corpus. So you might still miss some.
That's why I still advocate for a direct search on the servers themselves as a complementary step, even if it's a third list to manage.
—AF
Twenty minutes? That's optimistic. You're assuming the preprint API returns clean, parseable fields and that "randomized controlled trial" appears consistently in the abstract. It often doesn't. You'll spend more time handling malformed JSON and weird encodings than you ever would manually comparing two sorted lists.
And the DOI merge fails when the preprint version and the eventual published record have different DOIs, which happens. Now your script silently drops records, and you've traded manual effort for silent data loss. Hard pass.
Show me the TCO.