We’ve been running a pilot using Cohere’s Command R+ for an internal enterprise RAG system for about six months now. Our use case is querying a large corpus of technical documentation and process guides.
I’ve been quietly reading a lot of the comparisons here, and wanted to share our specific experience. The cost-to-performance ratio for our retrieval-heavy workflows has been good, especially the 128K context. We found its strength is in following complex instructions for synthesis, but it can be overly cautious in its responses.
My main question for others using it: how have you tuned the generation parameters for less “hedging” and more direct answers, while maintaining accuracy? We’re also evaluating if the newer, larger context models from other providers might change the calculus for us.
Interesting point about the cautious tone. We've seen that too, especially when dealing with documented but potentially conflicting procedures.
For your tuning question, we had some success being much more explicit in the system prompt about the style of answer we wanted. Phrases like "provide a definitive answer based solely on the provided context" and specifying "do not use hedging language such as 'it may be' or 'it is possible that'" helped more than adjusting the temperature or top_p parameters alone. It still errs on the side of safety, but it's less verbose about it.
On the larger context models from others, that's a good consideration. For us, the retrieval quality and the model's instruction-following for synthesis have been the bigger factors than raw context length, so we're sticking for now. The cost dynamics might shift, though.
Your point about explicit system prompts is well taken and mirrors our approach. We've found, however, that Command R+ can occasionally overcorrect when given strict stylistic directives, leading to a false certainty where it will state an incorrect detail from the context with high confidence. It's a trade-off between the cautious hedging and hallucinated precision.
We implemented a secondary verification step for high-stakes queries where the model is prompted to cite its source chunks and then a separate, simpler classifier checks for alignment between the answer and those citations. This has let us keep the system prompt directives strong while adding a safety net.
On your last point about retrieval quality being paramount, I fully agree. We've benchmarked against models with longer native context and found that for most queries, our dense retrieval pipeline feeding highly relevant chunks into Command R+ outperforms a model wading through 200k tokens of less-filtered text. The synthesis capability really is the differentiator.
— Harper
The hedging drove me nuts during our initial testing. We got better results by adding "assume the provided context is definitive" to the prompt. But like others said, it can swing too far the other way.
Have you looked at their playground settings? Lowering the temperature a notch actually helped us get more concise answers, surprisingly. It didn't fix the cautious tone, but made the responses tighter.
On the larger context models, isn't the retrieval quality more of a bottleneck than the model's window most of the time? That's been our hang-up.