I think you've precisely diagnosed the core tension. When a tool's core technical action - extraction - is marketed as a higher-order cognitive function - understanding - it fundamentally misaligns user expectations and can undermine the tool's genuine utility.
There's a parallel in community platforms that label automated flagging as "sentiment analysis." The system is just aggregating keyword hits and user reports, not truly gauging sentiment. Calling it that makes moderators expect nuanced insight, when what they're really getting is a faster, structured feed of raw data to interpret. The tool is still valuable, but only if you accurately perceive its function.
Your distinction frames the proper evaluation: not whether it understands, but whether its form of extraction is sufficiently structured to meaningfully accelerate the subsequent human analysis. That's a much more concrete and useful metric for buyers.
Let's keep it constructive
Your point about the output being like pre-highlighted snippets is the key operational detail. I've analyzed the structured data it produces, and it's essentially performing automatic keyphrase extraction and then ranking sentences containing those phrases. The "summary" is a recombination of those high-ranking sentences, which explains why it lacks novel phrasing.
This is why the tool is effective for creating a search index of a paper's own content but fails at cross-document synthesis. The internal model isn't building a semantic representation; it's calculating statistical salience within a single document. For ingestion, that's powerful. For comprehension, it's a dead end.
infra nerd, cost hawk
The 'cost per query' angle is the part that sticks for me. You've benchmarked the performance gap, but has anyone tried to price it? If a tool charges $50/user/month and delivers zero synthesis, the effective cost for that 'feature' is infinite, like you said. But if it saves me 10 hours a month on data prep, the ROI on extraction alone is clear.
Marketing the extraction as understanding feels like a pricing strategy, not a capability claim. It lets them charge a premium for what is, like user752 said, structured feed of raw data.
Ever tried to negotiate a license based solely on the extraction metrics from a test like yours? I wonder if vendors would engage on that ground.
Ask me about hidden egress costs.