Been tuning a RAG pipeline and got tired of stale knowledge bases. Built a system where "bad" LLM outputs (low confidence, user corrections) automatically trigger updates.
Core idea: When the LLM's answer is flagged or the user provides a correction, the system:
1. Validates the new info against a trusted source (in my case, a curated internal doc repo).
2. If valid, it creates a new document fragment.
3. Triggers a GitOps workflow to rebuild and redeploy the vector database.
Here's the heart of it. A Kubernetes `CronJob` and `Job` that watch for feedback events (from a logging stream) and process them:
```yaml
# feedback-processor-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: feedback-vector-updater
spec:
schedule: "*/15 * * * *" # Every 15 mins
jobTemplate:
spec:
template:
spec:
containers:
- name: processor
image: mycompany/feedback-ingestor:latest
env:
- name: FEEDBACK_LOG_PATH
value: "/var/log/feedback/events.log"
- name: VECTOR_STORE_API
value: "http://qdrant-service:6333"
volumes:
- name: feedback-log
persistentVolumeClaim:
claimName: feedback-log-pvc
```
The ingestor image is a simple Python script that:
* Parses the log for correction events.
* Fetches the relevant source doc to validate the update.
* Submits a Pull Request to the knowledge base Git repo with the new fragment.
Argo CD or Flux (I'm using Flux) picks up the PR merge and runs the embedding pipeline. The new vector index is rolled out via a canary on my k3s edge clusters. 🧠
Results? The KB used to drift after a few weeks. Now it self-heals. Most useful for product docs and internal runbooks. Still tweaking the validation step to avoid noise. Anyone else trying something similar?
yaml all the things