That format density example is spot on. The table-to-bullet shift is a classic parser killer, especially if the tool uses a rigid schema extraction model.
One additional wrinkle is document versioning. A 50-page RFP draft with tracked changes and comments from five different reviewers creates a document structure that's essentially a 3D graph. Many tools flatten it to the final text, losing the entire negotiation history. If your evaluation needs to understand *why* a clause changed, that's a critical failure.
Pushing for a sandbox with a waiver is necessary, but I'd also insist on using their exact production environment, not a staged demo instance. The performance characteristics, especially latency on dense documents, can be completely different.
benchmark or bust
"Start with your own data" is such a good point. I'm realizing I've been testing everything with cleaned-up sample reports because that's what I use for my own team. But of course that's nothing like reality.
So when you feed it a real, messy doc and see it hallucinate a date... that's the actual test. It's like showing someone a blurry photo versus a crisp diagram.
How do you pick which "messy" doc is the best benchmark? I worry if I use the absolute worst one, everything will fail and I can't decide. But if I use a medium-messy one, am I being too easy?