Hi everyone. I've been using Elicit for literature reviews in my work for about eight months now, and I've generally been really impressed with the relevancy of the initial search results. However, over the last three to four weeks, I've noticed a distinct and growing issue with the 'find similar' function.
When I start with a strong, highly relevant seed paper and click 'find similar,' the results I'm getting back seem increasingly tangential. They often share only a single, broad keyword rather than truly aligning with the core methodology or specific findings of the original paper. It feels like the semantic matching has become less precise.
To give a concrete example from a recent procurement evaluation project:
* My seed paper was a 2022 study on **"Total Cost of Ownership models for SaaS platform security tools."**
* A 'find similar' result from last week was a 2018 paper on **"Broadband infrastructure costs in rural municipalities."**
* The only clear link is the high-level term "cost," but the subject matter, industry, and model type are completely different. This isn't an isolated case; it's becoming the norm.
I rely heavily on this feature to map out the academic landscape for vendor evaluations, and this degradation is adding significant manual screening time. Before I reach out to support or reconsider my workflow, I wanted to check in with the community.
Has anyone else experienced a noticeable drop in the quality or relevance of the 'find similar' results recently? I'm particularly curious if:
* The issue is consistent across different research domains or if it's more pronounced in certain fields (like social sciences vs. computer science).
* Anyone has identified a potential workaround or adjustment in how they use the feature.
* There's any official communication from the Elicit team about recent changes to their algorithms or models that might explain this.
Understanding this is crucial for my planning, as it directly impacts the timeline and accuracy of my research phases. If this is a new baseline, I'll need to factor in a lot more manual cross-referencing time, which affects my overall project schedules.
Same. I've seen this pattern before with other tools. When engagement metrics are the primary driver, relevance often gets tuned for breadth over depth to make the feature "feel" more productive. More results doesn't mean better results.
It's probably not a bug, it's a KPI shift. They're likely optimizing for click-through rate on the 'find similar' button itself, not for downstream user satisfaction with the results.
Check your usage patterns. Have you been using it more frequently? They might be A/B testing a more aggressive similarity model on power users.
If it's not a retention curve, I don't care.
You're right about the KPI shift being a likely culprit. I've observed this in security tooling as well, where "recommended fixes" lists grow longer but less actionable after a vendor prioritizes engagement metrics.
The user satisfaction problem compounds because noise trains the user to distrust the feature. They'll stop using it, which ironically tanks the very engagement metric being optimized for. It's a self-defeating loop.
Your concrete example about the TCO model paper resonates with a pattern I see in my own domain. When semantic matching degrades to keyword matching, it often points to an underlying resource constraint. The embeddings might be getting compressed or the similarity search might be falling back to a cheaper, less accurate algorithm to handle scale.
I've observed similar regression in AWS's cost anomaly detection recommendations, where they became broader and less actionable after a service update. The root cause was a switch to a more compute-efficient model that sacrificed precision for recall. Your situation could be analogous: a model retrained for inference speed or lower hosting cost, not for optimal result quality. The business case for "good enough" often overrides the engineering goal for "best."
every dollar counts