Just wrapped up a 6-week pilot deploying Relevance AI for our internal Confluence knowledge base. We have about 200 users, mostly in support and engineering. The goal was to cut down on the "I can't find anything" emails and make our tribal knowledge more accessible.
The good: The setup was surprisingly smooth. The Confluence connector worked without a hitch, and the AI's answers feel more grounded in our actual docs than other tools we've tested. The chat interface is clean and users picked it up fast.
The lessons: The big one was tuning the search scope. By default, it was pulling from some very old, deprecated spaces and giving outdated answers. Had to spend a solid day in the dashboard configuring the source filters. Also, while the out-of-the-box answers are good, we found we needed to add a few custom "instructions" to shape the tone (e.g., "always mention the KB article title you're referencing"). For a team our size, the pricing felt fair, but you need to be mindful of those monthly query limits.
Overall, it's been a win for deflecting simple how-to questions. It's not a magic bullet—you still need clean source material—but as a smart search layer, it's pretty effective.
The search scope tuning is such a universal gotcha. We hit that exact issue with a HubSpot knowledge base project - it kept pulling from archived "sunset" articles and confusing everyone. Setting those source filters isn't hard, but you don't realize you need to do it until you get a glaringly wrong answer.
Your point on custom instructions is spot on. We added one like "if the answer involves a process, list the steps concisely" and it cut down on follow up questions. Did you have to do much prompting to get it to consistently cite those article titles?
It's not marketing, it's logic.
The citation piece is where a lot of these tools stumble out of the box. Relevance AI does a decent job pulling the title into the response, but I've found it sometimes truncates or paraphrases the title in confusing ways. We ended up adding a custom instruction like "Always include the full article title in a parenthetical at the end of the sentence where you reference it" and that cleaned things up significantly. Still catches maybe 90% of references now, but every now and then it'll drop a title entirely if the answer is blending content from three different pages.
One thing I'd add: we also had to explicitly tell it to prioritize fresher articles when titles overlapped (we have several "Onboarding Checklist v3, v4, v5" in our Confluence). Did you run into any versioning conflicts with your HubSpot archive, or was the sunset flagging enough?
The title truncation problem is real. We added a rule to "use the article's exact title, no shortening" but still saw occasional paraphrasing. I think the model sometimes attempts to make titles more conversational when they're embedded in a sentence.
Our versioning issue was with API docs, not checklists. We tagged articles with "latest_api_version: v2" and then added a system prompt to check that tag when answering technical questions. It helped, but required manual tagging of old pages.
Have you tried weighting by last modified date in the dashboard instead of just telling the AI to prioritize fresh content? I'm curious if the system-level filter is more reliable than a prompt.