I've been conducting a rather rigorous evaluation of Humata.ai for a potential document-heavy migration assessment project, precisely the kind of tool that promises to shortcut the initial analysis phase of legacy system documentation. However, I've hit a persistent and frankly confounding roadblock that undermines its core value proposition: the system frequently returns a **'No answer found'** response for queries where I have definitive, manual proof that the answer exists within the uploaded document.
This isn't a matter of ambiguous phrasing. I'm referring to direct, verbatim searches for concepts and data points that are clearly present. For instance, when analyzing a legacy application's data dictionary PDF (a 150-page monster, typical of the systems we often deal with), asking a specific question like *"What is the data type for the CUST_REF field in the ACCOUNT_MASTER table?"* yields 'No answer found.' A simple Ctrl+F in the source PDF confirms the string `CUST_REF VARCHAR(20)` is present. This is a fundamental failure.
My hypothesis, based on observed behavior, points to a few potential failure modes in its ingestion or query pipeline:
* **Chunking Strategy Pathology:** The document segmentation logic might be severing context. If the answer lies near a page break, diagram, or table, the semantic chunk may be truncated or orphaned, making it invisible to the query engine. This is a critical flaw for technical documentation where context is local and precise.
* **Overzealous Relevance Filtering:** The model's confidence threshold for returning an answer may be set prohibitively high. In an attempt to avoid hallucinations, it might be rejecting valid, lower-confidence matches that would, in fact, be correct.
* **Format Degradation:** Complex PDF layouts (two-column text, embedded tables, footnotes) are notorious for corrupting text extraction order. What we see as clear text is often a garbled sequence to the parser, breaking the model's ability to comprehend it.
Before I write this tool off as unsuitable for technical due diligence, has anyone else performed systematic tests and devised workarounds? I've attempted the usual:
* Re-uploading the document with OCR enabled (even for text-based PDFs).
* Reformulating the question multiple ways (from keyword-based to natural language).
* Breaking down the query into absurdly simple components.
The results are inconsistent at best. For a community focused on insights, this seems like a critical pitfall to document. If the tool cannot reliably answer direct, fact-based questions from its primary source, its utility for complex analysis—like understanding legacy system interdependencies pre-migration—becomes highly suspect. I'm leaning towards a more pragmatic, hybrid approach: using a brute-force text search for pinpoint data, and reserving LLM-based tools for higher-level synthesis *only after* the raw data is confirmed.
Plan for failure.
James K.