I've been conducting a systematic evaluation of NotebookLM's core feature—its ability to ground responses strictly in the provided source materials. The marketing emphasizes "grounded in your sources" to reduce hallucinations, a significant problem in standard RAG implementations. My hypothesis was that under stress, the system would either fail to ground properly or produce overly cautious, unhelpful responses.
I constructed a controlled test using a corpus of three technical research papers I am intimately familiar with (topics: transformer optimization, protein folding with ML, and federated learning architectures). I then asked a series of progressively more complex questions.
**Methodology:**
* **Source Docs:** Three PDFs, totaling ~85 pages.
* **Question Types:**
1. Direct extraction: "What was the learning rate used in Experiment 2 of Paper A?"
2. Synthesis across sources: "Compare the approaches to privacy preservation in Papers B and C."
3. Inferential leap: "Based on the limitations discussed across all papers, what would be a promising research direction?"
4. Introduced concept not in sources: "How do the findings in Paper A relate to the SwiGLU activation function?" (SwiGLU was never mentioned).
* **Evaluation Criteria:** Verbatim grounding, appropriate citation, handling of "unknown" information.
**Findings:**
1. **Direct extraction** performed adequately. Responses were correct and correctly cited the relevant source page. Latency was acceptable.
2. **Synthesis questions** revealed the first weakness. While the answers were generally "grounded" in that sentences could be traced to source snippets, the *connective tissue*—the analytical comparison—often felt generic and lacked depth. It was a summarization of relevant points side-by-side, not a true synthesis.
3. **The critical failure** occurred with the introduced concept (SwiGLU). Instead of stating the concept was not discussed in the sources, NotebookLM produced a plausible-sounding but entirely fabricated response. It claimed Paper A "employed the SwiGLU activation function to improve gradient flow," citing a specific page. That page discussed gradient clipping, not activation functions. This is a **critical hallucination**, precisely the issue "grounding" is meant to prevent.
**Conclusion from this test:** The grounding mechanism appears to be a strong *influence*, not a strict tether. Under pressure—when a query contains an unknown entity—the system seems to fall back on its base language model's parametric knowledge and generation tendencies, while *forcing* a citation to the sources. This is a dangerous failure mode because the incorrect information is presented with the veneer of source-based authority.
I intend to expand this stress test with a larger document set and more adversarial queries. For now, my skepticism is validated. Users should treat its citations with caution and not assume complete immunity from hallucinations. The system is best used for direct Q&A on source content, not for exploratory reasoning that might involve external concepts.
benchmarks or bust