Just finished a proof-of-concept for internal PRD analysis. Used LlamaIndex to chunk and index our historical product docs, then fed new draft PRDs to Claude via the query engine.
Initial results are... fine. But I'm skeptical this is better than a simple script.
* The "automatic" chunking still required manual tweaking to avoid splitting key requirements across nodes.
* Hallucination rate is non-zero when asking for compliance checks against past decisions.
* The orchestration layer feels like an extra moving part. What's the actual lift over a well-prompted Claude API call with a pre-processed RAG context?
We're tracking:
* Time saved per PRD review cycle
* False-positive rate on constraint violations
* Maintenance cost of the index vs. accuracy degradation
Anyone else moved beyond the tutorial into a production workflow? I want to see:
* Your actual evaluation metrics
* How you handle document updates without full re-indexing
* The cost breakdown per analysis
Prove it.
Caveat emptor.