I've been conducting a systematic literature review for a forthcoming paper on eBPF-based security observability in Kubernetes, and like many here, I've incorporated ResearchRabbit into my workflow for discovery and visualization. While the "Similar Work" and "forward snowballing" features are reasonably effective, I've found the "backward snowballing" (identifying prior papers that cite a given seed paper) to be significantly less reliable than a manual, rigorous search on Google Scholar or a dedicated library database.
My process typically begins with a set of 5-7 highly relevant seed papers. When I add one, such as "{{Seccub: Automating SBOMs for Cloud-Native Applications}}" (a fictional example for this post), and trigger backward snowballing, the results are often incomplete. The algorithm appears to prioritize a subset of citing papers, potentially based on its own relevance scoring, rather than presenting a comprehensive list. In several instances, I have manually discovered 15-20 citing works on Google Scholar, while ResearchRabbit returned only 8-10, missing key papers that were foundational to different branches of the research tree.
The core issue seems to be in the data sourcing and completeness. A robust backward snowballing requires a complete and well-structured citation index. My hypothesis, based on working with academic APIs, is that ResearchRabbit might be relying on a single or limited set of data sources (e.g., Semantic Scholar, OpenAlex) which, while good, are not exhaustive. Manual checking across multiple sources still yields a more complete picture.
**Concrete Example from My Current Project:**
Seed Paper: *"FlowScope: Microservice Call Graph Reconstruction via Distributed Tracing"*
* ResearchRabbit Backward Snowballing returned **12** papers.
* Manual Google Scholar "Cited by" search returned **19** papers.
* The 7 missing papers included two critical works that applied the core concept to serverless environments, a direct link to my research interest.
This discrepancy forces a hybrid workflow, negating some of the promised efficiency. The cost, in terms of time, is non-trivial when you're dealing with dozens of seed papers across a large review.
**Suggested Mitigations & Questions for the Community:**
* Are others observing similar gaps, particularly in fast-moving fields like cloud infra?
* Has anyone developed a semi-automated method to cross-validate ResearchRabbit's backward results? I've considered a simple script to take a DOI and check it against the Crossref or OpenAlex APIs directly, but haven't implemented it yet.
* For critical paths in a literature review, I now default to manual verification. This seems to be a necessary step for academic rigor, which unfortunately reduces the tool's value proposition for this specific feature.
The visualizations and collaborative features remain useful, but for the core academic function of tracing a citation graph backward in time, I cannot currently rely on it as a single source of truth.
That's such a good point, and it really hits on the promise versus reality of these tools. I've seen the same gap with backward snowballing. My guess is they're using a curated index rather than a full scholarly scrape, and they're filtering for "influential" citations to keep the signal-to-noise ratio high for most users. But for a systematic review, noise *is* the signal - you need the complete set.
I still start with Rabbit for the graph and forward snowballing, but I treat its backward results as a starting point, not a definitive list. It's a bummer we still need that manual Scholar check as a backup for thoroughness. Have you found any other tool that does better with this specific task? I'm starting to think it's a fundamental data problem.
good docs save lives
That aligns with what I've seen too, though my work is more applied. In customer success, we use tools that promise a "complete view" but often filter based on internal scoring. It creates a gap between the tool's logic and what you actually need to see.
I'm curious, when you say "the algorithm appears to prioritize a subset," have you noticed any pattern to what it does surface? Is it recency, or journal prestige? If the misses are predictable, maybe there's a way to adjust the seed paper selection to compensate.