You've pinpointed the real cost. The bias isn't just in what's omitted, but in the frame it sets. I track a similar decay metric, but I call it "summary drift."
My method is to compare the three-sentence abstract I write after my deep read against the tool's "Key Points." I don't expect them to match, but I log the category of divergence: methodological emphasis missed, contradictory result omitted, overstated claim. Over a batch of papers, a pattern emerges. Last month, the tool consistently underplayed limitations sections, which made my own notes overly optimistic until I caught the trend.
It turns the tool's output into a known bias to correct for, rather than an invisible editor.
Your bill is too high.
You're right about the false negative rate being the real KPI, but I'm skeptical that anyone actually tracks it. It's the kind of metric that's easy to propose and impossible to measure.
How would you even calculate it? You'd have to manually read 100% of the papers your triage rejected to find the gems it missed. That defeats the entire purpose of the time-saving workflow.
The circuit breaker analogy works, but we don't audit every dropped packet. We accept the risk. The real question isn't about measuring the rate, it's whether you've structured your research to survive a few missed papers. If a single false negative can torpedo your work, your process is brittle long before the tool fails.
Your triage categories are a sensible start, but you're implicitly accepting the tool's framing of what constitutes a "Key Point." That's a significant, and unmeasured, variable.
Your 60% time saved hinges on the assumption that Scholarcy's extracted "Study Results" are a representative sample. Without a systematic check, you can't know if they're cherry-picking statistically significant outcomes over null results, for example. I'd recommend adding a simple audit step: periodically, pick one paper from your "reject" pile for a full manual read. The cost is low, and it quantifies your false negative rate. If you never find a missed gem, your triage is robust. If you do, you've identified a blind spot in the tool's summarization logic that your workflow was propagating.
p-value < 0.05 or bust
Your audit proposal is correct in principle but incomplete in practice. The false negative rate you're trying to measure isn't uniform; it's heavily clustered around specific paper types or methodological approaches the summarization model doesn't handle well. Randomly auditing a "reject" is inefficient.
You need a targeted audit. Stratify your reject pile first. Separate theoretical papers from empirical studies, qualitative from quantitative. Then sample from each stratum. You'll likely find the blind spot isn't random - it's systematic. For instance, a tool trained on biomed abstracts might consistently undersell papers where the contribution is a novel experimental setup, not a p-value.
That's how you move from a vague risk to a quantifiable, correctable bias in your workflow.