Hello everyone. I've been conducting a deep-dive evaluation of Elicit for a client considering it for their literature review workflows, and a recurring theme in our internal discussions has been the opacity surrounding the underlying language model. I wanted to share my structured concerns and see if the community has gathered any concrete intelligence.
From a procurement and vendor evaluation standpoint, the specific model used is a critical component of the "Technical Architecture & Roadmap" pillar. Not knowing this creates several material risks for an organization:
* **Performance Boundary Forecasting:** Without knowing the base model (e.g., GPT-4, Claude 3, a fine-tuned open-source model), it's challenging to predict its inherent strengths and limitations. Is it exceptionally strong at reasoning but weaker at strict adherence to prompt instructions? This affects how we design internal user guidelines.
* **Cost and Stability Modeling:** The model is a primary cost driver for the vendor. Changes to it (e.g., a switch from one major provider to another) could directly impact pricing tiers, performance SLAs, and even feature availability down the line. Our contract negotiation levers are weaker without this insight.
* **Data Governance and Compliance:** Certain regulated industries need to map data flows, including the AI models involved. "Proprietary blend" is often insufficient for legal and security review questionnaires. It becomes a stumbling block in the procurement process.
In my vendor scoring framework, I typically allocate points for transparency in this area. Elicit seems to position itself more as an end-to-end *application*, which is valid, but for enterprise procurement, the components matter. My current workaround has been to infer capabilities through systematic testing against known model benchmarks, but this is resource-intensive.
I'm curious if any of you, through direct inquiry or usage patterns, have pinned down more specifics:
* Have you received a definitive answer from their sales or support engineering teams on the model lineage?
* Does the observed behavior in complex query chains point strongly to a particular family of models?
* How are you addressing this transparency gap in your own vendor evaluations or risk assessments?
Understanding this is key to building a long-term vendor relationship, as model shifts can feel like a completely different product if not managed carefully.
null
Performance forecasting is one thing, but you're missing the bigger liability. If you don't know the model, you can't assess its training data for licensing or copyright risks. Your client could be building a workflow on top of something with a legally ambiguous foundation.
The cost angle is valid, but model changes also introduce security and compliance blind spots. A switch from a vendor-hosted model to an open-source variant could radically alter the data privacy posture, and you wouldn't see it coming until the release notes. Your SLA might cover uptime, but it probably doesn't cover that.
Has your client asked them point-blank in an RFP? Any vendor that dodges that isn't just being opaque, they're telling you they don't want to be held accountable.