I've been conducting an in-depth evaluation of SciSpace (formerly Typeset) for the past three months, primarily focusing on its recommendation engine and "My Feed" functionality. My initial hypothesis was that the platform could significantly streamline literature discovery for my ongoing research in experimental design for digital products. However, the signal-to-noise ratio in the feed has degraded to the point of being non-functional, prompting a systematic investigation into its failure modes.
The core issue is the persistent suggestion of papers that are tangentially related at best, based on superficial keyword matching rather than semantic understanding or citation graph proximity. For example, my primary interest is in **Bayesian A/B testing methodologies**. Despite this, my feed is consistently populated with:
* Papers on Bayesian networks in computational biology.
* Articles on clinical trial design for pharmaceutical applications, simply because they share the word "trial."
* Pre-prints on entirely unrelated statistical methods that happen to cite a foundational paper I've saved, with no weighting for the context of that citation.
This suggests a recommendation algorithm that is likely operating on a brittle set of heuristics. To diagnose the problem, I attempted to "train" the feed through explicit feedback, a process analogous to calibrating a machine learning model. My actions included:
1. **Meticulous profile curation:** Filling out all research interest fields, linking my ORCID, and ensuring my published work was accurately reflected.
2. **Active engagement logging:** Systematically upvoting ("Relevant") and downvoting ("Not Relevant") on every suggested paper for a two-week period.
3. **Source diversification:** Following specific authors, journals, and curated collections known for quality in my field.
The results were statistically insignificant. The rate of irrelevant suggestions did not show a measurable decrease week-over-week. A simple binomial test would confirm that the feedback mechanism is not leading to a meaningful improvement (p-value >> 0.05).
My current working theory is that the system suffers from one or more of the following algorithmic flaws:
* **Over-reliance on TF-IDF style keyword extraction** without sufficient stemming or entity recognition (e.g., distinguishing "test" in a statistical context from "test" in a software engineering context).
* **A lack of temporal decay** in its model, causing it to perpetually recommend papers based on a single, early-saved article rather than adapting to my evolving, recent interactions.
* **Poor integration of the citation graph**, where it counts a citation but doesn't weigh the strength of connection or the relevance of the citing paper's domain.
Has anyone else performed a similar analysis or found a configuration workflow that actually works? I'm particularly interested in whether the "Exclude from recommendations" function has a measurable impact, or if there are hidden URL parameters or API endpoints that allow for more granular control. Sharing any longitudinal data on your feed's precision/recall would be invaluable for comparison.
p-value < 0.05 or bust
Yeah, that keyword matching problem sounds all too familiar. It's like the system sees "Bayesian" and just throws the entire kitchen sink at you without understanding the specific domain. I've found these feeds often fail because they're not great at picking up on negative signals.
Have you tried explicitly marking those irrelevant papers as "not interested"? Sometimes that can train the filter over a few weeks, but it's a slow process. The citation graph issue you mentioned is the real killer though - if it can't weight the *reason* for a citation, the whole thing falls apart.
ian
The "not interested" button is supposed to do that, right? But I'm not sure if it actually works, or if it's just there to make us feel better. How long does it take for the feedback to actually change the algorithm? Has anyone seen it work reliably?
I guess my worry is that marking things "not interested" could accidentally filter out a paper I'd want, just because it uses a broad keyword. If the system doesn't understand context, my negative feedback might be too blunt.
Your worry about the negative feedback being too blunt is exactly right. I've seen this happen with other "smart" feeds. The problem is, these algorithms often treat a "not interested" signal as a veto against a *topic tag*, not the nuanced reason why that specific paper was irrelevant.
So you could end up blacklisting the keyword "Bayesian" for a few weeks because it keeps surfacing papers on Bayesian *philosophy* instead of your A/B testing focus. Good luck getting those relevant method papers to show up again.
As for reliability, I've never seen a single instance where marking a paper "not interested" produced a visible change in under 48 hours. Usually it takes a week and a batch of similar feedback. They're almost certainly aggregating user signals to retrain a model on a schedule, not making real-time adjustments.
Cloud costs are not destiny.
Thanks for sharing your detailed evaluation. That systematic breakdown really helps clarify the failure mode - it sounds like the algorithm is stuck in a keyword matching loop without the contextual layer it needs.
You're touching on a key tension in these systems. If the recommendation engine can't distinguish between the *semantic role* of a shared keyword or citation, then "Bayesian A/B testing" and "Bayesian networks in biology" become neighbors. It's a fundamental design hurdle.
Your last point about citation context weighting is spot on. A citation in a methods section versus a literature review carries completely different signals. Have you noticed if the platform allows you to seed your feed with specific *papers* versus just broad topics? Sometimes that can anchor the system a bit better, at least in theory.