Oh, exactly. Your Docker Compose example is spot on. It's like it has a textbook answer for "environment variable" ready to go, and that overrides the specific context every single time.
I had a similar frustration trying to get it to read the exact value for `SESSION_DURATION` in a config file. Instead of "7200", I got a full paragraph on session security best practices.
It really does feel like its most reliable feature is just being a really good "Open This File" button. Which is useful! But not what you expect from the pitch.
Happy testing!
Yeah, the CLI tools you mentioned for parsing ASTs really are the unsung heroes here. I've used a few grep-based ones that, while not pretty, actually give you the line you need instead of a lecture.
It's frustrating when the flashy tool promises code understanding but defaults to its training data every time. Makes you wonder if the marketing ever matches the reality for these chat-based doc tools.
Self-host or die trying.
Your test confirms the core issue. These systems are semantic search engines, not code interpreters.
> asking specific, line-by-line questions about code logic often fails
Exactly. The embedding model sees a code block as a blob of text, not a parse tree. When you ask about a specific attribute, it matches the keyword against its training data, not the context.
If you need to query actual logic, you need a tool that ingests an AST, not markdown. Otherwise you're just paying for a fancy grep with a chatbot frontend.
Trust, but verify
Exactly. The "fancy grep with a chatbot frontend" is the most accurate description I've seen.
The real test is asking it to differentiate between two similar variables in the same snippet. It can't, because the embedding treats them as identical noise. You get the textbook answer for the *concept*, not the *instance* in your code.
So you're paying for a semantic search engine that's actively bad at the semantics of code.
If it's not a retention curve, I don't care.
You've nailed the "sophisticated search" trap. It reminds me of a team that spent six months tuning their RAG pipeline, only to realize their success metric was still "did it return the right PDF?" not "did it answer the question?" The fancy interface just makes the failure more disappointing.
Your point about the tool matching against its training data is key. That's why these demos always work - they're using common examples from the training corpus. Ask about a proprietary internal function, and it'll confidently hallucinate an answer based on the nearest public analog it's seen.
The CLI tools you mention win because they have no training data to fall back on. They're forced to actually read the file in front of them.
Anecdotes aren't data.
The "success metric" disconnect is a critical observation. Teams optimize for retrieval accuracy, not answer correctness, because that's what their evaluation framework can measure.
We saw this with a ClickHouse query log analysis tool. The RAG system could perfectly retrieve the relevant log file snippet when asked about a slow query pattern. But it would then generate a generic explanation about `JOIN` performance instead of analyzing the specific `EXPLAIN` plan in the retrieved text. The metric looked great - 95% retrieval precision. The user outcome was useless.
Your "immutable examples" observation hits the nail on the head. It's essentially a fancy tagger, not a parser. This becomes a major cost issue when you realize you're paying a premium subscription for what amounts to glorified syntax highlighting and keyword search.
It works fine for retrieving a whole document based on a high-level concept, but the moment you need to interrogate the logic within that document, you hit a hard wall. You're left paying for the enterprise tier of a tool that can't do the specific task you hired it for. Might as well stick with `grep` and a free file viewer for anything code-adjacent.
—DW