I've been conducting a systematic literature review for a risk assessment framework, and while Elicit's core literature search functionality is robust, I find myself consistently perplexed by the concept matrix presentation. The intent—to provide a synthesized, at-a-glance comparison of key claims and topics across multiple papers—is academically sound and aligns with rigorous review methodologies. However, the execution feels like it introduces more cognitive load than it resolves.
My primary point of confusion stems from the automatic extraction and categorization of concepts. The algorithm appears to select and phrase concepts in a manner that is often too granular or, conversely, overly broad, missing the nuanced thesis of the paper in question. This leads to a matrix where:
* The rows (papers) are accurate, but the columns (concepts) contain extracted sentences that are out of context, requiring me to open the paper to understand the true framing.
* Significant, recurring themes across my result set are sometimes omitted from the matrix entirely, while minor, tangential mentions are elevated to concept status.
* The lack of user-defined control over the initial concept generation forces a workflow where I must first review the matrix, identify its shortcomings, and then mentally re-map the papers against my own, predefined criteria.
From a compliance and audit perspective, this is problematic. When documenting the literature review section of a security framework or preparing an audit rationale, I require clear, traceable linkages between source material and the derived control or risk statement. An automatically generated matrix that misrepresents or obscures the central argument of a source undermines the audit trail.
I am curious if others have developed effective workflows to mitigate this. Do you rely on the matrix as a starting point for manual correction, or do you bypass it entirely in favor of a more traditional annotation method? Furthermore, has anyone successfully used the "synthesize" feature in conjunction with the matrix to produce a reliable summary, or does the confusion in the source matrix propagate into the synthesis?
—at