Tutorial: Using AI to audit a knowledge base for outdated information

Interesting approach. I've applied similar logic to stale documentation in BI tools, where outdated data source credentials or deprecated metric definitions cause dashboard failures.

The temporal keyword scan works well, but you might consider adding structured metadata to future-proof this. Tag each KB entry with a 'last reviewed' date and a 'review cadence' (monthly, quarterly, annually) at creation. You can then automate the hotlist generation using a simple query against that metadata table, which is more reliable than parsing free text for date patterns.

This also creates an audit trail for compliance. The AI method is great for the initial cleanup of legacy content, but a lightweight tagging system prevents the problem from recurring at scale.

So you're paying Notion's AI tax just to run a glorified find-in-page search for date strings? You can get that for free with grep and a basic regex. The real issue isn't finding "Q1 2023," it's knowing whether the underlying procedure is still valid, which an LLM hallucinating in your Notion sidebar absolutely cannot tell you.

This feels like a solution that introduces more risk than it solves. You're outsourcing the initial triage to a black box that might miss crucial context or, worse, misinterpret it. I'd rather trust a junior analyst with a checklist than a probabilistic word guesser that I can't hold accountable.

You're right that grep is cheaper, but it's about scope. A regex can catch "Q1 2023", but it won't flag two pages with conflicting gateway instructions unless they share identical outdated strings. That's the triage use case - narrowing 10k pages to 50 likely candidates for human review.

The hallucination risk is real, which is why the original post said to use it only for generating a hotlist, not making the final call. A junior analyst with a checklist is better, but they still need to know which pages to give them.

The point about conflict detection is valid. I've seen this in Terraform modules where one team's VPC module references an outdated AMI ID while another's uses the new one. A regex won't catch that unless the strings match. An LLM can sometimes spot the semantic mismatch.

But calling it "triage" doesn't absolve the method. You still need to validate the model's output, and that validation step is nontrivial. If the AI gives you 50 pages with "potential conflicts," you now have to manually verify each flagged item to see if it's a real conflict, a hallucination, or a misunderstanding of nuance. That's a new workload.

My issue is the false confidence. It's not just about hallucinations. It's about the model's complete lack of institutional context. A page might say "legacy gateway" because the procedure is for migrating *off* it, which is correct and current. The AI will flag it as outdated, wasting time. You've traded a broad search problem for a precision validation problem, and I'm not convinced the trade-off is positive for anything beyond the most trivial date-string hunt.

Exactly. The "new workload" is the hidden cost nobody factors in. You're not saving analyst hours, you're just shifting them from scanning to debugging the AI's output.

And the institutional context problem is fatal. An LLM trained on generic web text has zero clue about your internal migration projects or which deprecated terms are still in active, correct use.

So you spend more time writing prompts to explain your company's nuances to the model than you'd spend just having the analyst look at the docs in the first place. The triage argument falls apart when the signal-to-noise ratio is terrible.

You've put your finger on the crucial economic question here - is it a net time save or just a task transfer? In my experience, this breaks along a clear line: the initial, massive cleanup versus ongoing maintenance.

For that first-pass audit of thousands of legacy pages, an AI scan can be a net positive even with the debugging overhead. Manually scanning for temporal phrases and potential conflicts across a vast, unloved KB is soul-crushing and error-prone work. If the tool cuts the candidate list from 2000 pages to 200, you've still saved a huge amount of initial eyeball time, even if you have to scrutinize those 200.

But you're absolutely right for the steady state. Using an LLM for monthly reviews of a clean, curated knowledge base? That's where the "new workload" of prompt engineering and output validation likely outweighs the benefit. The triage argument only holds water when the haystack is genuinely massive.

The institutional context gap is the real ceiling, though. You can't prompt-in your company's entire history.

"Faster than manually re-reading" is optimistic. You're trading a slow, known process for a fast, unknown one.

I've seen these hotlists. They're full of false positives from pages that say "as of last year's policy" or "legacy system X (now deprecated)." Now your human has to parse those qualifiers anyway. The triage didn't save work, it just changed the search terms.

If your docs are in Markdown, you already have structure. A simple script to flag any date string older than X months would be more reliable. At least you can see the logic.

> "You're trading a slow, known process for a fast, unknown one."

That's the core of it, isn't it? I think the value of that trade depends heavily on the existing state of your data. If your KB is already in clean Markdown with consistent date metadata, you're right, a script is the way. But in my last gig, our internal docs were a mix of Confluence exports, Slack threads dumped to text, and PDFs. There was no structure to query.

In that mess, a regex for dates would have missed a ton. An LLM's noise was, paradoxically, a better starting point than nothing. The false positives were a new task, but sifting through 200 messy pages was still less daunting than starting with 2000 completely unsorted ones.

Maybe the real takeaway is to fix the structure first. But that's another huge project nobody wants to fund.

The hallucination point is valid, but grep won't even get you to the starting line on the real problem. You can't regex for conflicting advice or subtle procedural drift. So you're left choosing between a junior analyst's slow, literal checklist and a model's fast, creative mistakes.

I'll take the creative mistakes for the first pass through a dumpster fire of a knowledge base. At least it might spot the contradiction between the "deprecated" page and the "new standard" page that both still work. A junior analyst following a checklist will just tick the box next to each one.

The junior analyst with a checklist is a known quantity, which is critical for risk assessment. Their process is repeatable and accountable. You can audit their work and pinpoint where a step was missed.

Choosing the "creative mistakes" of a model introduces an unquantifiable variable. You might spot a contradiction between deprecated and new pages, but you could also miss ten others due to the model's inherent lack of grounding. In vendor risk terms, you're swapping a predictable, high-effort provider for a cheap, innovative one with no SOC 2 report and a history of hallucinations. The latter creates more work in validation and dispute resolution than it saves in discovery.

The economic calculation changes if, and only if, you treat the AI output as a non-deterministic sampling method, not a triage tool. It's a way to get a random, possibly insightful slice of the mess to inform how you build your actual, rule-based script.

Your method has merit for the initial triage, but you're ignoring the API cost. Notion AI isn't free, and if your KB is large, you're looking at a non trivial bill for a task that's fundamentally pattern matching.

For the "scan for temporal keywords" phase, you can get 90% of the value with a simple `grep -E "Q[1-4] 20(1[0-9]|2[0-3])|last year|version [0-9]"` on your Markdown export. Zero hallucinations, zero vendor lock in, and you can pipe the results into a spreadsheet instantly.

Save the LLM spend for the contradiction check, where it might actually add value a regex can't. Even then, you need a tight, repeatable prompt, otherwise the cost per page adds up fast.

That's a clever way to use Notion's built-in AI. I've found the "contradiction check" prompt especially useful for team onboarding docs. You can feed it the new hire guide and the detailed technical setup guide and ask it to find mismatches in permissions or required software versions. It flagged that one guide mentioned a tool we'd already sunsetted.

Just watch out for cost if your KB is huge. Starting with a grep for date patterns to narrow the set before you send anything to the AI can keep those API calls (or credits) down.

> "a triage tool, not the auditor"

Correct. The risk is teams stop there. You get a hotlist, but it doesn't flag what the model *didn't* find.

Bigger liability is outdated configurations the AI sees as correct because it's syntactically fine. Think firewall rule references or deprecated API endpoints still written as valid. No temporal keyword, no direct contradiction, just silently wrong.

Human verification needs a threat model, not just the AI's output.

That's a fair point about grep being more reliable when you have clean Markdown. I've seen scripts like that work beautifully for teams with good doc hygiene.

But you're absolutely right that it just changes the search terms. The real work starts when you have to interpret the list. If your script flags a page from 2021, you still need a human to open it and ask: is this procedure still true? That's where the time goes, not in generating the list.

Maybe the real win is combining the two? Use the cheap, reliable grep for date strings to get your initial list, then use a human (or an LLM if you must) to ask the specific question "does this outdated date actually matter to the current procedure?" for each hit.