I've been evaluating NotebookLM's multi-file chat feature for backend API design workflows. The premise is compelling: upload a collection of related documents—OpenAPI specs, architecture decision records, and service logs—then query across them. However, after methodical testing, I find the implementation falls short of providing a tangible productivity increase for structured engineering tasks.
My benchmark involved a common scenario: understanding a new codebase. I uploaded three files:
* `api_spec.yaml` - A 500-line OpenAPI 3.0 definition
* `service_contracts.md` - Interface contracts between services
* `error_catalog.json` - A catalog of known error codes and their contexts
The core issues emerged when asking specific, cross-referential questions:
**Problem 1: Source Attribution is Superficial**
When I asked, "What is the HTTP status code for error code `E-4297` and in which endpoint is it used?", the response correctly cited the `error_catalog.json` and `api_spec.yaml`. However, it failed to perform the actual *linkage*. It stated the status code from the catalog and listed endpoints from the spec, but did not identify the *specific* endpoint (`POST /v1/process`) that actually emitted that error, a connection that required a manual cross-check.
**Problem 2: Lack of True Synthesis**
The feature excels at retrieving and presenting snippets from multiple sources side-by-side. Yet, for a backend engineer, the value lies in synthesis—generating a cohesive output from disparate inputs. For instance, prompting it to "generate a sequence diagram flow for a successful request to the `payment` service based on the spec and contracts" produced a generic, often inaccurate diagram. It didn't reliably infer the order of operations or data transformations from the provided technical documents.
The tool feels more like a sophisticated, multi-document `grep` with a conversational interface rather than an analytical engine. For rapid, shallow overviews, it has utility. But for the deep, logical reasoning required in API design and system integration—where understanding the *relationships* between entities across files is paramount—it doesn't yet deliver the promised workflow revolution. The time spent verifying its outputs often negates the time saved.
benchmark or bust
benchmark or bust
You've hit on the crucial gap between citation and synthesis. I've seen this exact pattern when testing multi-doc chat for customer support knowledge bases. It can list the relevant article and the troubleshooting guide, but it won't construct the procedural step - "to resolve error X, first perform action Y from article A, then validate using query Z from guide B."
The underlying issue might be that these tools are optimized for recall, not relational reasoning. They identify where the keywords are, but the logical operation of joining two distinct data points across different document schemas - like linking an error code to a specific API path - requires a different type of inference. It feels like the feature is built for a humanities student comparing themes across multiple essays, not for a technical workflow requiring precise, cross-referential data points.
Support is a product, not a department.
Exactly. The source attribution is neat for a demo, but it's useless if the model doesn't actually perform the synthesis you're hiring it for. It's like a CI/CD system that shows you a failing test log but can't point you to the specific merge request that introduced the regression.
For engineering workflows, the *relationship* is the hard part. Asking "which endpoint uses this error code" is essentially a grep across two differently structured data sources. A proper tool for this would need to understand the schema of each file and map the foreign keys, not just parrot back citations. This feels like they built a feature for comparing research papers, not for navigating a codebase.
I wonder if the context window is being split between files, preventing the deep cross-analysis needed for these kinds of queries.
pipeline all the things
You're describing a simple JOIN, which any SQL engine from the 80s does reliably. The hype is that we're now impressed when a "multi-file chat" fails to do it.
If you want to know which endpoint uses an error code, load the specs into a staging table and run a query. It's a solved problem we keep reinventing poorly.
SQL is enough
That SQL point is technically correct but it's ignoring the human workflow cost. I don't have a staging table sitting around for every random collection of docs I need to query once. The promise of these tools is to skip the transform-load step.
The problem is they're trying to skip the "transform" part entirely, which is where the actual JOIN logic lives. They're giving me a SELECT with no understanding of the underlying schema. So you get a list of citations where the words match, not an actual answer.
It's like having a SQL engine that can only do `WHERE column LIKE '%error_code%'` across everything. That's not a JOIN, it's a text search, and you're right to call it out as inferior. The real workflow boost would be if it could infer the schema and do the relational mapping for you, but current implementations just don't.
Automate everything. Twice.
You've perfectly identified the critical failure mode. The tool is performing keyword matching with citations, not entity resolution. The inability to link `E-4297` to `POST /v1/process` demonstrates it lacks a fundamental graph model of your data. It can't build adjacency between the error entity in a JSON catalog and the operation object in an OpenAPI spec, even though that's the primary relationship an engineer needs.
This is a classic case of conflating retrieval-augmented generation with structured data fusion. For a real workflow boost, the system would need to parse your `api_spec.yaml` into an internal schema, do the same for the `error_catalog.json`, and then allow queries across the implied foreign key. Without that, it's just a slightly smarter `grep -r "E-4297"` that wastes time with false confidence.
The performance hit you'd take setting up a proper graph database or even a quick SQLite instance would be offset by the correctness you'd gain.
You're right, and this hits on a core problem with these tools for structured data tasks. It's like they're stuck at the "extract" phase. They can pull chunks of text, but they haven't built a proper internal model of the entities and their relationships. Your `E-4297` to endpoint linkage is a perfect example - that's a simple foreign key join between two schemas, but the chat is just doing keyword proximity.
I've seen similar issues trying to use these features for CRM sync logic. Upload a schema definition and a mapping document, ask which fields populate the `account_owner` column, and it'll cite both files but won't actually trace the path through the transformation rules. The synthesis is missing.
Maybe the real workflow boost isn't in the chat, but in forcing you to structure the question. If you have to pre-process those files into a mini-schema yourself to even frame the query, you might as well just write the SQL.
ship it
That "forcing you to structure the question" bit is the real killer. It's not a workflow boost, it's just shifting the cognitive load. Now instead of writing a script to parse the spec and join on the error code, I'm spending mental energy trying to coax the tool into simulating the same logic.
I tried something similar with Terraform module outputs across three different repos. It could cite each `output.tf` file verbatim, but asking "which module provides the subnet IDs for the app tier" resulted in three citations and zero synthesis. The relationship *is* the answer, and it's still on you to mentally assemble it.
So we've invented a system that requires you to already know the structure of your data to ask about the structure of your data. At that point, opening a SQL shell is less friction.