I've been evaluating Humata for our internal technical documentation, which is a mix of architecture overviews, API specs, and a *lot* of code snippets (Terraform modules, Kubernetes manifests, Lambda functions). The promise of querying docs in plain English is great, but I was skeptical about how it would parse actual code.
My initial tests have been... mixed. Here's what I've found:
**What works well:**
* It correctly identifies the programming language in most snippets (tags them as Python, YAML, HCL, etc.).
* If you ask a high-level question like "What's the authentication method shown in the Python example?", it can usually pull the right answer from the comments or surrounding prose.
* Summarization of a document that contains code generally stays accurate to the conceptual flow.
**Where it gets tricky:**
* Asking specific, line-by-line questions about code logic often fails. For example, "In the provided Terraform code, why is `create_before_destroy` set to true?" might yield a generic explanation of the lifecycle meta-argument instead of referencing the specific resource block.
* It seems to treat code blocks more as immutable examples rather than analyzable text. Asking it to "refactor the following snippet for cost optimization" on a Lambda function yielded a very generic AWS tip, not a modified version of the code I provided.
* Cross-referencing between multiple snippets in a doc is limited. If one snippet defines a variable and another uses it, Humata doesn't always make that connection.
My current take is that it's useful for navigating *to* the right document or section containing the code you need, but you still have to read and interpret the snippets yourself. It's more of a powerful Ctrl+F than a code-analysis assistant.
Has anyone else pushed it further with technical docs? Found any prompts or document structures that make it handle code more effectively?
-- Amy
Cloud cost nerd. No, I don't use Reserved Instances.
Yeah, that tracks with what I've seen. The high-level stuff works because it's leaning on the surrounding text. Where it breaks down is exactly on those line-by-line technical specifics.
I ran into this with CloudFormation templates. Asking "what's the DependsOn attribute for the Lambda function in this snippet?" would often just parrot the AWS documentation on DependsOn itself, completely ignoring the actual code block.
It feels like the model is trained to recognize code as a type of content block, but not to truly parse its structure for detailed Q&A. For now, it's a decent documentation search, but a poor code reviewer.
Totally agree on the high-level vs. line-by-line gap. I've noticed the same when asking about specific parameters in our K8s manifests. It'll often give me the generic Kubernetes docs definition of a field instead of referencing the actual value I've set in the snippet.
You mentioning Terraform examples hits home. I tried asking "what's the CIDR block for the VPC in the third module?" and it just explained what a CIDR block is. It's like the code is scenery, not interactive text. Makes me wonder if it's tokenizing those blocks differently, maybe truncating them?