Just tried the new Gemini-powered NotebookLM update. The accuracy feels... better? But I'm not 100% sold yet.
Here's my quick test with a project brief PDF:
* **Factual recall:** Spot on for dates and names from the doc. Big improvement.
* **Summarizing:** Still great, maybe faster.
* **The big "but":** When I asked for a risk analysis based on the doc's data, it still felt like it *inferred* a bit too much. Got the general idea right, but added one "risk" that wasn't actually implied in the source.
Anyone else running tests? Specifically:
* Are you finding fewer hallucinations with your own source material?
* Is it sticking closer to the text when you ask for analysis?
* How does it compare to other tools you use for doc interrogation?
Would love to compare notes!
— Jason
Let's build better workflows.
Thanks for sharing this test, Jason. I'm in a similar boat. I've been feeding it some Python library docs and my own project notes, and I've noticed the exact same pattern you described.
The factual recall is definitely sharper, which is a huge help. But like you said, on analysis tasks, it still seems to wander a bit from a strict reading. It's almost like it gets a little too eager to provide a complete answer and fills in small gaps with its own assumptions. That one extra "risk" it added for you is a perfect example of what I'm seeing.
Have you noticed if the tendency to infer changes based on the length or complexity of the source document? I'm wondering if it's more prone to sticking to the text with shorter, more focused sources.
still learning
That's a really good question about document length and complexity. From my own tests with some lengthy API spec PDFs versus concise meeting notes, I haven't found a clear correlation. If anything, shorter sources might give it *less* to work with, making those inferential leaps more tempting to "complete" the answer.
I've had better luck by being hyper-specific in my prompts. Instead of "do a risk analysis," I'll write "list only the risks explicitly mentioned in the third section." That usually locks it down. It's a bit more work, but it feels like guiding the model's focus is the key, regardless of source size.
How are you structuring your prompts when you use your Python docs? Do you find yourself needing to steer it more than you'd like?
Latency is the enemy, but consistency is the goal.