Hi everyone. I’m helping a small research team (just 4 of us) get started with our first systematic review. The volume of papers is already overwhelming, and we’re looking for a tool to help with the screening and data extraction stages.
I’ve seen Scholarcy recommended in a few places for summarizing articles and pulling out key facts. The demo looks promising, but I’m having a hard time finding real-world examples of teams using it *throughout* an actual systematic review process, not just for reading single papers.
My main questions are:
- If you use it for production reviews, how does it fit into your workflow? Do you use it right after database searches, or later?
- How reliable is the automatic data extraction for things like sample sizes, outcomes, and key findings? We’re worried about missing something critical if we rely on it too much.
- For a small team on a tight budget, does the subscription feel worth it compared to managing with a combination of, say, Rayyan and a lot of manual work?
I’d be really grateful for any honest insights. We’re trying to be efficient but don’t want to invest time and money into a tool that might not hold up for the rigorous needs of a systematic review. 😅
Small team, big decisions
We tried to integrate it at the screening stage last year. It falls apart quickly with any decent volume.
The automatic extraction is decent for a quick skim to understand a paper's main argument. For systematic review data points, it's not reliable enough to trust. We found it would miss specific outcome measures or misreport sample sizes if the wording in the paper was atypical. You'll spend as much time verifying its output as you would extracting manually.
For a team of four, I'd stick with Rayyan for screening and build a solid extraction form in something like Google Sheets or Airtable. Scholarcy's subscription becomes another tool you have to manage, not a solution. The cost versus the manual verification work doesn't balance out.
Your CRM is lying to you.
I've seen a similar pattern when teams try to automate extraction too early. The reliability issue with sample sizes and outcomes is a known constraint.
You might consider a hybrid approach if you proceed: use Scholarcy's summary to *accelerate* your initial screening in Rayyan, especially for title/abstract triage. But treat its extracted data points as candidate values only, requiring mandatory verification in your extraction form. We measured this once; it cut screening time by about 30% but saved almost no time in the final data synthesis stage because everything needed a human check.
For a team of four on a tight budget, that marginal time savings in one phase likely doesn't justify the subscription cost. You'd be better off investing in a well-structured, pre-piloted extraction template in your sheets or Airtable setup.