The sales pitch was compelling: an AI research assistant to map our niche field. The reality was a pricey wrapper for keyword search with extra steps.
Our main gripes:
- The 'full-text analysis' couldn't parse methodology sections in our own uploaded PDFs. It kept highlighting generic terms.
- The cost per seat was justified by 'workflow tools' that were just shared folders and basic tags. Our self-hosted Zotero instance does more.
- No clear exit strategy. Exporting our 'enhanced' metadata was a mess of JSON files. Felt like data hostage-taking to justify renewal.
We asked for a detailed breakdown of what the AI was actually doing versus a simple semantic search library. The answer was marketing speak about 'proprietary engines.' For that price, we expected auditability, not a black box on a hypetrain.
Doubt everything
We ran a trial for our marketing team with similar issues. The cost per seat only made sense if the tool actually saved analyst time, but it didn't. It just repackaged what we could already do with Google Scholar alerts and some manual tagging.
>Felt like data hostage-taking
This is key. We ended up with a folder of useless JSON. Did you ever get a straight answer on what their proprietary engine actually adds?
The export issue worries me. I'm looking at a few tools like this for our team, and that lock-in risk is real.
When you asked about their proprietary engine, did you get any specific metrics on recall or precision compared to regular semantic search? Or was it just a "trust us, it's smarter" kind of response?
It seems like the real cost isn't just the seat price, but the time lost cleaning up the export if you need to leave.
Exactly. We ran into the same "black box" issue when evaluating a different tool last quarter. They couldn't show any actual traceability from input to output.
If they can't provide a simple benchmark against a basic TF-IDF or even a public model like SPECTER, then the 'proprietary engine' claim is just a pricing tactic.
You're right to expect auditability. For that cost, you should be able to see confidence scores and the model's provenance. Anything less is just renting a keyword search with a fancy UI.
Ship it, but test it first