Everyone's rushing to slap an "AI" label on their support portal, and the demos always look flawless. I decided to see what the friction points actually are when you try to build a simple, self-hosted FAQ bot using Relevance AI's platform and your own internal docs. The promise is straightforward: ingest your documentation, get a chatbot that answers questions. The reality, as usual, has a few more edges.
I used a PDF export of our internal security policy wiki (about 50 pages covering everything from password rules to incident response) as the test knowledge base. The upload process itself is simple, I'll give them that. The real questions start after ingestion. You have to configure the "workflow," which is their visual builder for the bot's logic. The critical part is the grounding—making sure it actually uses your provided docs and doesn't just hallucinate an answer from its base model. I found the default similarity search settings needed tightening; initially, it was pulling in irrelevant snippets from unrelated sections. Tuning the chunk size and overlap isn't for the faint of heart.
Then there's the audit trail, or lack thereof. For a compliance-heavy use case, you need to know *which* source document snippets were used to generate an answer, and you need that logged. Out of the box, it's not a persistent, tamper-evident log—it's more of a developer preview in the workflow run. I had to add extra steps to capture and pipe that data to our own logging system. Without that, this is a toy, not a tool for anything involving GDPR right of explanation.
The final bot worked... mostly. For straightforward policy questions like "What is the minimum password length?" it was accurate. But for anything requiring synthesis—like "What steps should we take if a laptop is stolen?"—it would occasionally conflate steps from different procedures or miss crucial ordering. It's a decent first-pass retrieval engine, but calling it an "agent" feels generous. The cost of running these workflows with high-volume queries also adds up quickly; it's not just the model, it's the compute for the search and the logic steps.
Bottom line: It can work for a simple, low-stakes FAQ where occasional inaccuracy is tolerable. But if you're in a regulated industry or need verifiable answers, plan to invest significant time in prompt tuning, grounding validation, and building a proper audit log system around it. Another silver bullet that turns out to be lead.
—Greg
Trust but verify