Guide: Building a quick 'hallucination detector' for sales-agent summaries.

Sales agents are feeding meeting notes into LLMs and getting back dangerously polished summaries. The core risk is fabricated details—pricing, timelines, features—that get committed to CRM. You need a fast, automated check before that data propagates.

Here's a pragmatic, retrieval-augmented method. It compares the generated summary against the raw source text (transcript/notes). The principle is simple: key factual claims must be grounded in source material.

**Implementation Steps**

1. **Chunk & Embed Source:** Split your source document into sentences or small chunks. Generate embeddings (use a local model like `all-MiniLM-L6-v2` for speed).
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
source_chunks = ["Chunk 1 text...", "Chunk 2 text..."]
source_embeddings = model.encode(source_chunks)
```

2. **Extract Claims from Summary:** Use a lightweight LLM call (e.g., GPT-3.5-turbo) to decompose the summary into discrete factual statements.
```
Prompt: "List every factual claim in the following text, one per line. Include only claims about products, pricing, dates, commitments, or specifications."
```

3. **Semantic Search & Score:** For each claim, find the most similar source chunk via cosine similarity. If the top similarity score is below a threshold (e.g., 0.7), flag the claim as potentially hallucinated.
```python
import numpy as np
claim_embedding = model.encode([claim])
similarities = np.dot(source_embeddings, claim_embedding.T).flatten()
top_score = similarities.max()
if top_score < threshold:
flagged_claims.append(claim)
```

**Key Tuning Points**
* Set the similarity threshold based on your domain language. Start at 0.65 and adjust.
* Pre-process text: normalize dates, numbers, product names.
* This catches fabrications, not omissions. It's a sanity check, not a full audit.

-dk