Just built a 'dumb question' bot for my team to avoid interrupting deep work. It's a hit.

I've seen at least three teams in my org try to build "quick question" bots over the last year. Most of them died within a month because the answers were too generic, too slow, or just plain wrong—ending up as more of a distraction than the interruptions they were meant to prevent. So when I saw your thread title, my immediate reaction was skepticism. Making a bot that actually reduces interruptions instead of creating a new notification sink is a non-trivial engineering problem.

I need to see the implementation details to believe this is a sustainable solution and not just a novelty that will wear off. Specifically:

* **Model choice & prompting:** What model(s) are you routing to on Poe? Are you using a single bot or a combination? The difference between using a fine-tuned model like Claude-3-Opus for reasoning versus a cheaper, faster model like Llama-3-70B-Instruct for simple lookups is massive in terms of cost and latency. Show me the actual system prompt. I've benchmarked "answer concisely" against multi-shot examples with explicit guardrails, and the performance delta is significant.
* **Context management:** How are you feeding it team-specific context? If someone asks "where's the staging deploy docs?" or "what's the format for the quarterly review PR?", the bot needs internal knowledge. Are you using Poe's document upload feature, or are you preprocessing context into the prompt? If the latter, what's your chunking and retrieval strategy? The naive "stuff the last 10 Confluence pages into the context window" approach falls apart under real usage.
* **Evaluation & failure modes:** What metrics are you tracking to confirm it's a "hit"? Just subjective "people say they like it," or actual logs showing decreased Slack pings for certain question types? More importantly, what's your protocol for when it hallucinates an answer about a build process or gives outdated API info? A single confident wrong answer can destroy trust permanently.

Here's a snippet of the prompt template I used in a similar, now-archived, experiment. The failure was usually in the retrieval step, not the LLM itself.

```markdown
You are a team assistant for the Data Platform team. Your ONLY knowledge sources are:
1. The provided context snippets below.
2. General programming and software engineering knowledge.

If the question is about team-specific processes, tools, or code, and the answer is NOT VERIFIABLE in the context, you MUST say: "I don't have enough specific context to answer that reliably. Please check the team wiki or ask in #platform-help."

Context for this query:

{{retrieved_documents}}

Question: {{user_input}}
```
The bot became a bottleneck because maintaining the vector store was a manual process. Is your solution actually scalable, or is it just a clever wrapper on a Poe bot that will degrade as your internal knowledge evolves?

I'm interested in the architecture because if you've cracked this, it's more valuable than most of the custom chat interfaces I've benchmarked. But I need concrete evidence, not just enthusiasm.

/benchmark