I've been running Scholarcy through its paces with a stack of recent white papers and technical briefs. The summary cards are handy, I'll grant them that. But as I'm looking at the extracted 'key facts' or methodology steps, I'm constantly left with the same nagging question: how sure are you about this?
The tool presents these extractions with the same flat, authoritative tone whether it's pulling a clear, explicit figure from a table or making a somewhat interpretive leap from a dense paragraph of jargon. For instance, in a recent analysis of a cloud cost-optimization case study, it confidently spat out a specific percentage for infrastructure savings. When I went back to the source, that number was *implied* across several sections but never stated as a single, neat figure. Scholarcy had, charitably, inferred it.
Would it be so hard to attach a simple, underlying confidence score? Not some AI black box explanation, just a heuristic. High confidence for direct quotes, numbers in tables, or sentences matching clear patterns. Lower confidence for paraphrased summaries of complex arguments. Even a three-tier system (High/Medium/Low) based on extraction clarity would be a game-changer. It would tell me when to trust the summary as a true timesaver and when I need to put my skeptic's hat on and dive into the source text myself.
Right now, we're asked to take the output entirely on faith. And in my line of work, faith is not a viable strategy. Anyone else feel like they're playing a guessing game with the algorithm's certainty?
cg
cg