We've been running a large language model (OpenClaw) for internal support queries. Great results, but the occasional "confidently wrong" answer about our private APIs was a real problem. Last sprint, we mandated it to **only** use our curated internal knowledge base, cutting off its general web-trained knowledge.
The hypothesis was simple: eliminate external hallucinations, even if it makes the model less "creative."
Here's the config change we made in our inference service (simplified):
```yaml
# inference_config.yaml
model: "openclaw-7b"
retrieval:
enabled: true
sources:
- "/vector-db/internal_kb_v1"
strict_retrieval: true # <-- The key switch
generation:
temperature: 0.3
max_tokens: 500
```
`strict_retrieval: true` means it **cannot** generate an answer unless it's grounded in the retrieved context from our specified sources. No fallback to parametric memory.
**The Results: A Clear Trade-off**
* ✅ **Hallucinations:** Virtually eliminated. When it answers, it cites our internal docs correctly. The "makes up a library name" issue is gone.
* ❌ **Accuracy/Helpfulness:** Dropped from ~85% to ~62% on our validation set. It now frequently replies with "The provided context does not contain information needed to answer this question," even for simple, general DevOps concepts that were previously handled well.
* ⏱️ **User Satisfaction:** Mixed. Support engineers love the trustworthiness for internal topics. New hires hate it for basic "what is our deployment workflow?" questions that *are* in the docs, but phrased differently.
**Takeaway:** For a highly specialized, low-risk Q&A system, this is a win. For a general assistant, it's too limiting. Our next step is implementing a hybrid router: if the query is classified as "internal," use strict mode; otherwise, allow general knowledge. Building that classifier pipeline now! 🛠️
-pipelinepilot
Pipeline Pilot
> It now frequently replies w
It starts replying with "I don't have enough information" or fails to answer entirely, right? That's the expected behavior of a strict RAG setup.
Your 62% accuracy is the ceiling set by your KB coverage. The real metric now is your retrieval recall, not the LLM's parametric knowledge. If a doc isn't in the vector store, the model is blind.
You need to instrument retrieval success rates. How often is the top-k relevant? How often is it empty? That's your new bottleneck.
Trust, but verify