Streaming responses from an LlamaIndex query engine is great for UX, but I kept hitting a wall: the citations would either be missing from the chunked stream or arrive only at the very end, breaking the user's trust. If you're deploying this in a production setting where source attribution is non-negotiable, that's a real problem.
After wrestling with a few approaches, here's the pragmatic workflow that finally worked for me. The key is to handle the citation extraction *asynchronously* from the token stream itself.
* **Use the `streaming=True` flag** on your query engine, but don't just iterate over the response_gen.
* **Intercept the `Response` object** from the `astream()` generator. The `source_nodes` are already attached there, but you need to pull them out before the text tokens are yielded.
* **Map and deduplicate on the fly**. For each text chunk, I check the `source_nodes` and map them using the node's start char index. This lets me emit a `[Citation: 1]` marker inline as soon as the relevant text is streamed, not five seconds later.
The main pitfalls I ran into:
* **Duplicate citations**: The same node can be relevant to multiple sequential chunks. You need a simple cache to avoid spamming `[Citation: 1, Citation: 1]`.
* **Buffer management**: For very short text chunks, you might need to buffer slightly to ensure the citation marker is placed at a natural break (e.g., end of a sentence) rather than mid-word.
* **Context window bleed**: Be cautious with your chunk sizes during indexing. If your source nodes are too large, the citation will be attached to a huge block of text, reducing its utility.
In the end, this added about 15% more code to my streaming handler, but the result is a response that feels authoritative and verifiable from the first sentence. The procurement team loved it because it directly addressed their audit trail requirement without breaking the conversational flow.
stay pragmatic
Another layer of complexity to manage for the sake of live citations? Seems like you're just moving the problem.
The real issue is that these "smart" retrieval systems still need you to write a bunch of bespoke glue code to get basic functionality. You've built a citation mapper, but what happens when the node's start char index is off by a few tokens? Now your inline marker is in the wrong place.
Debugging this async mapping will eat more time than just sending the sources at the end. Sometimes waiting five seconds is the correct engineering trade-off.
-- old school
You're right that it adds complexity, but waiting until the end for citations undermines the entire point of streaming in a user-facing context. It breaks the trust you're trying to build.
The mapping logic doesn't need to be perfect to be useful. Even if a citation marker is slightly off, showing the user that sources are being woven in live is a different class of experience than a big delay followed by a footnote dump. It signals transparency as it happens.
- GG