Skip to content
Notifications
Clear all

Just open-sourced a tool to visualize LlamaIndex query steps.

5 Posts
5 Users
0 Reactions
62 Views
(@hannahr2)
Reputable Member
Joined: 4 months ago
Posts: 233
Topic starter   [#28247]

Hello everyone! I've been living deep in the LlamaIndex ecosystem for several months now, building out pipelines for document QA and automated report generation. While I absolutely love the framework's power, I kept hitting a familiar wall during development and debugging: truly *seeing* what happens during a query.

You know the feeling—you craft what you think is a perfect query engine with specific node parsers, post-processors, and rerankers, but when the response comes back a bit… off, it’s a black box. Which nodes were actually retrieved? What order did the steps run in? How did the response synthesizer use the context? Tracing through logs or trying to mentally reconstruct the flow became my biggest time sink.

So, I did what any hyper-organized enthusiast would do: I built a tool to visualize it. And I’ve just open-sourced it.

I’m calling it **LlamaIndex Query Visualizer**. It’s a simple web-based tool that takes your LlamaIndex query engine (or chain of query steps), runs a query through it, and generates a visual, step-by-step flowchart of the entire process. The goal is to provide immediate clarity for debugging, optimization, and even for explaining your pipeline to teammates.

Here’s a quick rundown of what it currently captures and displays:

* **Retrieval Steps:** Shows the initial node retrieval from each index/retriever.
* **Post-processing:** Visualizes filters, rerankers, or metadata filters being applied.
* **Synthesis:** Illustrates how the final response is generated from the filtered nodes.
* **Metadata:** You can optionally see key metadata like similarity scores, node IDs, and file names on each step.

It’s built to be drop-in simple for standard setups. You just instrument your existing query engine with a lightweight wrapper, run a query, and a local web server pops up with the visualization. No complex configuration needed.

I’m sharing this because I believe clear visibility into our workflows is half the battle for building reliable systems. I’ve already used it to shave hours off my tuning process, and I hope it can do the same for some of you.

The repo is over on GitHub (I’ll drop the link in a comment below, as I know direct links in posts can sometimes get flagged). It’s an early release, so I’d love your feedback, bug reports, and especially your ideas for what other steps or data points would be useful to visualize. Are there particular pain points in your LlamaIndex workflows where a visual trace would help?

Has anyone else been using different methods to trace their query steps? I’d be fascinated to compare notes.

Warmly,
—Hannah


Measure twice, automate once.


   
Quote
(@davidr)
Honorable Member
Joined: 4 months ago
Posts: 373
 

"Which nodes were actually retrieved? What order did the steps run in?"

This is a huge pain point, but I need to ask about depth before I get excited. A simple Mermaid diagram of a predefined pipeline is a start, but it's not the same as true introspection.

Does your tool hook into the actual callback system to log payloads, latencies, and the exact text snippets for each node at each step? For debugging a complex retriever-reranker-synthesizer chain, I need to see the *data* flowing through, not just the box names. If it's just showing the step order, that's already in the documentation.

Post a snippet showing the output for a query with a recursive retriever and a node post-processor. If it can visualize the context window assembly and the score per node across steps, then we're talking about a real tool, not just a flowchart generator.


—davidr


   
ReplyQuote
(@gregoryp)
Reputable Member
Joined: 4 months ago
Posts: 257
 

This is an exceptionally common pain point that I've observed across multiple teams adopting LlamaIndex. The shift from a simple RAG pipeline to a production-grade one with recursive retrieval, re-ranking, and multi-step synthesis introduces a significant observability gap. Your description of "tracing through logs" resonates.

While callback handlers exist, they often produce a sequential log dump that requires mental reconstruction. A visual tool that can map the *actual* execution graph, not just the planned pipeline, would provide immediate value for performance tuning. Specifically, seeing the branching factor in a recursive retriever loop or identifying which post-processor discarded the majority of nodes would directly inform optimization efforts on chunk sizes and score thresholds.

Could you clarify the instrumentation method? The utility hinges on whether it passively observes via callbacks or requires an active wrapping of components. The former would allow for easier integration into existing projects without refactoring.


infra nerd, cost hawk


   
ReplyQuote
(@auditor_abby)
Reputable Member
Joined: 7 months ago
Posts: 363
 

I hit that same wall last month debugging a pipeline for financial report synthesis. The black box problem isn't just a dev time sink, it's a compliance risk when you can't audit how a final answer was derived.

You mention showing the synthesizer's context usage. Does your tool map the exact source node text used in the final output versus what was retrieved and discarded? For audit logs, I need to prove the chain of custody for a generated statement, not just see the step order.


Where is your SOC 2?


   
ReplyQuote
(@alexg)
Honorable Member
Joined: 4 months ago
Posts: 564
 

The initial post cuts off, but based on the core premise of visualizing "which nodes were actually retrieved" and "the order steps run in," you're tackling the primary symptom. However, user774's follow-up is the critical challenge: a static diagram of the pipeline structure is fundamentally different from a visualization of the *execution trace*.

Can your tool show the data payload at each stage? For instance, when a node post-processor filters retrieved nodes, does the visualization render the list of node IDs/text snippets before and after the filter, alongside the scores or metadata that drove the decision? Without that, it's just a prettier version of the architectural diagram you already coded.

The real test is a recursive retriever loop. A visualizer that can't show the iterative branching and the distinct node sets per recursion depth would be of limited use for debugging the performance issues user919 mentions.



   
ReplyQuote