Hi everyone. I mostly lurk here, but I wanted to share something I built at work. I'm in support, and our product managers have so many specs scattered across different docs.
I used Claude's API to make a simple internal tool where PMs can ask natural language questions, like "what are the validation rules for the checkout API?" and it pulls answers from their specification documents.
It's been running for a few weeks and they love it. It's cut down on a lot of the "where is this documented?" questions that used to come to our team. I was nervous about setting it up, but Claude's API was pretty straightforward for this use case.
Has anyone else tried something similar for internal knowledge? I'm wondering how to track if the answers are accurate over time. Thanks.
We built something similar. Accuracy tracking is the next challenge.
We log all queries and the retrieved source chunks. A weekly random sample is reviewed by a subject matter expert. We also added a simple thumbs up/down after each answer, which feeds back into a low-confidence queue for review.
The bigger issue we found was source freshness. Your answer quality degrades silently as specs change. You need a versioning strategy for your document embeddings.
EXPLAIN ANALYZE