I've recently concluded a procurement process for a systematic review platform for our research department, and my findings have led me to a conclusion that seems to run counter to current trends. While Elicit's AI-driven literature discovery is impressive for exploratory scoping, our controlled evaluation showed it is not yet a replacement for a dedicated SR tool like Covidence when protocol adherence is paramount.
Our evaluation criteria, derived from PRISMA and Cochrane guidelines, focused on:
* **Audit Trail & Reproducibility:** Covidence provides a complete, step-by-step record of screening decisions, conflicts, and reasons for exclusion. Elicit's AI filtering, while fast, operates as a "black box" for the end-user. You cannot fully audit why a specific paper was omitted from its suggested list.
* **Structured Manual Search Upload & De-duplication:** Our workflow requires searching multiple databases (PubMed, Embase, etc.) with complex, validated syntax. Covidence's manual upload and robust de-duplication is seamless. Elicit is designed for its own semantic search; while you can upload a list, it's not the core function and the de-duplication logic is not as transparent.
* **Granular Customization of Screening Forms:** For data extraction, we need fields tailored to our PICO framework. Covidence allows for completely customizable forms. Elicit's extraction is fixed to its AI model's predefined categories, which can miss niche data points critical to our analysis.
The critical pitfall emerged during our pilot. Using an existing review topic as a benchmark, Elicit's AI search failed to surface three key studies that were retrieved by our traditional, exhaustive database search strings. These were not obscure papers; they were central to the field. The AI's ranking and selection algorithm had effectively hidden them. This is an unacceptable vendor risk for systematic review integrity.
My position is this: Elicit is a powerful tool for idea generation and rapid literature mapping. However, for a *systematic* review where missing a single study can bias the entire outcome, the controlled, manual, and fully auditable process facilitated by Covidence—starting with a comprehensive manual search—remains the methodologically sound choice. The "AI search + screening" model introduces an uncontrolled variable at the most critical phase. I am interested if other procurement or research operations professionals have conducted similar comparative validations.
Buyer beware, but with a spreadsheet.