Showcase: Built an internal knowledge Q&A chain with summarization and citation agents.

I've seen a lot of teams try to bolt a Q&A system onto their existing docs and wikis, expecting a ChatGPT-like experience but ending up with hallucinations and dead ends. The promise is great: ask a question in plain English and get a precise, cited answer from your internal documentation. The reality is usually a half-baked RAG pipeline that fails when you need actual nuance.

So I built a proof-of-concept using AutoGen to enforce a strict process and, crucially, to add a layer of summarization and citation verification *before* any answer reaches the user. The goal was to make it useful for engineers asking about internal APIs, deployment procedures, or post-mortem findings.

The core architecture uses three specialized agents orchestrated by a `GroupChat`:

1. **Retrieval Agent:** Its only job is to query the vector store (I used Chroma with all-MiniLM-L6-v2 embeddings). It receives the question and returns a list of relevant document chunks with metadata (source file, page number, etc.). It doesn't interpret, it just fetches.
2. **Analysis & Citation Agent:** This is the critical quality gate. It receives the raw chunks and the original question. Its instructions are to:
* Determine if the retrieved context is *sufficient and relevant* to answer the question.
* If not, request a new search with refined terms.
* If sufficient, synthesize a direct answer.
* **MANDATORY:** Extract exact quotes from the provided chunks that support each part of the answer and format the citations inline.
3. **Summarization & User-Facing Agent:** This agent receives the cited answer from the Analysis agent. Its role is to reformat the output for final consumption, ensuring clarity and conciseness, and to provide a final, clean summary with citations listed at the end.

Here's a simplified version of the key agent configurations. The system prompts are where you enforce the hard rules.

```python
# Retrieval Agent - Just fetches.
retrieval_agent = ConversableAgent(
name="retrieval_agent",
system_message="You are a retrieval tool. Given a user question, you will output ONLY a JSON list of relevant document chunks with 'text', 'source', and 'page' keys. Do not answer the question yourself. Do not add commentary.",
llm_config={"config_list": [{"model": "gpt-4"}]},
human_input_mode="NEVER",
)

# Analysis & Citation Agent - The enforcer.
analysis_agent = ConversableAgent(
name="analysis_agent",
system_message="""Your task is to answer ONLY using the provided context chunks.
1. Evaluate if the chunks directly contain the answer. If no, ask retrieval_agent for better chunks.
2. If yes, compose an answer.
3. **YOU MUST CITATION-TAG EVERY CLAIM.** Format citations as [source_file.pdf, page X].
Example: "The API rate limit is 1000 requests/hour [api_spec_v2.md, page 12]."
4. Pass your final cited answer to the summarization_agent.""",
llm_config={"config_list": [{"model": "gpt-4"}]},
human_input_mode="NEVER",
)

# Summarization Agent - Polishes for end-user.
summarization_agent = ConversableAgent(
name="summarization_agent",
system_message="You format the analysis agent's answer for final delivery. Ensure it is clear and concise. Compile a final list of all citations used. Do not add new information or citations.",
llm_config={"config_list": [{"model": "gpt-4"}]},
human_input_mode="NEVER",
)
```

**The Pain Points & Pitfalls:**

* **Cost & Latency:** This is a multi-LLM call chain. A single query can easily involve 3-4 GPT-4 calls (retrieval judgment, synthesis, summarization). You need caching and possibly a cheaper model for the retrieval agent.
* **Orchestration Overhead:** Getting the handoffs right and preventing agents from overstepping their roles requires very precise system prompts. Debugging a misbehaving agent chain is frustrating.
* **Citation Integrity:** The system is only as good as your chunking and embedding. If the retrieval agent pulls wrong or fragmented chunks, the analysis agent will struggle. You need a solid RAG foundation first.
* **Tool Integration:** I had to wrap the actual vector store query in a function and register it as a tool for the retrieval agent. AutoGen's tool handling works, but it adds another layer of configuration.

**Was it worth it?** For a high-stakes internal tool where accuracy is non-negotiable, yes. The separation of concerns—retrieval, verification/citation, presentation—drastically reduced hallucinations. For casual use, it's probably overkill. The main value was forcing a rigorous, auditable process between question and answer, which most naive RAG implementations completely lack.

If you're trying something similar, start by nailing your document preprocessing and embedding strategy. Then introduce AutoGen to harden the QA loop. Don't do it the other way around.

---