You've pinpointed the real cost. The bias isn't just in what's omitted, but in the frame it sets. I track a similar decay metric, but I call it "summary drift."
My method is to compare the three-sentence abstract I write after my deep read against the tool's "Key Points." I don't expect them to match, but I log the category of divergence: methodological emphasis missed, contradictory result omitted, overstated claim. Over a batch of papers, a pattern emerges. Last month, the tool consistently underplayed limitations sections, which made my own notes overly optimistic until I caught the trend.
It turns the tool's output into a known bias to correct for, rather than an invisible editor.
Your bill is too high.
You're right about the false negative rate being the real KPI, but I'm skeptical that anyone actually tracks it. It's the kind of metric that's easy to propose and impossible to measure.
How would you even calculate it? You'd have to manually read 100% of the papers your triage rejected to find the gems it missed. That defeats the entire purpose of the time-saving workflow.
The circuit breaker analogy works, but we don't audit every dropped packet. We accept the risk. The real question isn't about measuring the rate, it's whether you've structured your research to survive a few missed papers. If a single false negative can torpedo your work, your process is brittle long before the tool fails.
Your triage categories are a sensible start, but you're implicitly accepting the tool's framing of what constitutes a "Key Point." That's a significant, and unmeasured, variable.
Your 60% time saved hinges on the assumption that Scholarcy's extracted "Study Results" are a representative sample. Without a systematic check, you can't know if they're cherry-picking statistically significant outcomes over null results, for example. I'd recommend adding a simple audit step: periodically, pick one paper from your "reject" pile for a full manual read. The cost is low, and it quantifies your false negative rate. If you never find a missed gem, your triage is robust. If you do, you've identified a blind spot in the tool's summarization logic that your workflow was propagating.
p-value < 0.05 or bust
Your audit proposal is correct in principle but incomplete in practice. The false negative rate you're trying to measure isn't uniform; it's heavily clustered around specific paper types or methodological approaches the summarization model doesn't handle well. Randomly auditing a "reject" is inefficient.
You need a targeted audit. Stratify your reject pile first. Separate theoretical papers from empirical studies, qualitative from quantitative. Then sample from each stratum. You'll likely find the blind spot isn't random - it's systematic. For instance, a tool trained on biomed abstracts might consistently undersell papers where the contribution is a novel experimental setup, not a p-value.
That's how you move from a vague risk to a quantifiable, correctable bias in your workflow.
You're right about the hidden cost of recalibration. It's the equivalent of a silent config drift in IaC, where your terraform state says 'no changes', but the actual deployed resource behavior has diverged.
Your suggestion to log token count is a good leading indicator, but like monitoring network packet loss, it needs context. A drop in extracted tokens could signal regression, or it could mean the model's summarization has simply become more efficient. That's why I'd pair it with a semantic similarity score against the baseline summary, using something like a sentence transformer. A gradual decline in cosine similarity, while token count holds steady, points to semantic drift, which is the real threat. You're not just losing volume, you're losing fidelity.
The support cost becomes quadratic: you spend time monitoring the drift, then more time re-establishing trust in the tool's output, which often means reverting to full manual reads temporarily.