I've been evaluating Consensus alongside other models (Claude 3 Opus, GPT-4, open-source options) for a specific use case: generating structured forecasts from a mixed input of historical data and qualitative reports. The marketing suggests it excels at "grounded" outputs, but I'm interested in empirical accuracy, not just citation quality.
My preliminary testing protocol involves:
* **Input:** A standardized prompt containing a time-series dataset (e.g., last 8 quarters of sales) and a paragraph of analyst commentary.
* **Task:** Output a JSON with point forecasts and 80% prediction intervals for the next 4 quarters.
* **Evaluation:** Compare against held-out actuals using Mean Absolute Percentage Error (MAPE) and assess whether the true values fall within the predicted intervals (calibration check).
Initial results are mixed. While Consensus is less prone to "hallucinating" numbers wildly outside the data range, its forecasting accuracy is not consistently superior. For instance, on one test:
```json
// Consensus Output (simplified)
{
"forecasts": [
{"period": "Q1", "point": 1250, "lower_bound": 1180, "upper_bound": 1320},
{"period": "Q2", "point": 1280, "lower_bound": 1205, "upper_bound": 1355}
]
}
// Actuals: Q1=1210, Q2=1295.
// MAPE: ~1.7% (good), but Q1 actual was below the provided lower bound (poor calibration).
```
The core question: **Has anyone conducted rigorous, quantitative evaluations of Consensus specifically for forecasting or numerical prediction tasks?** I'm particularly interested in:
* Benchmarks comparing its accuracy against other frontier models on time-series extrapolation.
* Any observed systematic biases (e.g., conservatism, over-reliance on recent trends).
* Whether its "grounding" mechanism actually improves accuracy when noisy textual context is provided, or if it just adds latency and cost.
I've found plenty of demos for its RAG capabilities, but few published evaluations on its pure predictive performance. My hypothesis is that for numerical forecasting, a simpler, cheaper model fine-tuned on time-series data might outperform it significantly on a cost/accuracy basis. Looking for evidence to confirm or refute this.
Your methodology is sound, especially the focus on interval calibration. That's often where these models fail catastrophically. In my own structured testing for capacity planning (which is forecasting cloud spend), I found similar results. The "grounded" nature seemed to primarily suppress extreme outliers, but it didn't improve the core statistical accuracy of the point forecasts or the reliability of the confidence intervals.
Where I saw a marginal benefit was in handling the qualitative commentary. When the analyst text contained conflicting signals, Consensus was slightly better at not latching onto a single dramatic phrase and ignoring the rest, compared to a base GPT-4 model. However, for pure time-series extrapolation from clean historical data, a simple ARIMA model in a Python script consistently outperformed all the LLMs on both MAPE and interval hit rate.
Your truncated JSON output suggests it's producing very narrow bands. I'd be curious if your calibration check shows those 80% intervals are actually hitting anywhere near 80% of the held-out data, or if they're significantly overconfident.
every dollar counts
Interesting focus on the interval calibration. That's the key practical metric for any risk-adjusted decision. A forecast with bad intervals is worse than useless.
I ran a similar test for AWS budget forecasting using three years of monthly spend plus text from our internal planning docs. Consensus intervals were actually too narrow, missing the true value more than 20% of the time despite being 80% intervals. The point forecasts were fine, but the overconfidence made the output risky to use for actual procurement.
What's the compute cost per forecast run for you? If the accuracy delta is marginal, the ROI over a simple fine-tuned open-source model might not justify it.
Ask me about hidden egress costs.
That's a great point about overconfidence being more dangerous than just being wrong. Your result on narrow intervals is what I'd expect from any model that's essentially doing sophisticated pattern matching on historical data - they don't really grasp "unknown unknowns" that create tail risks.
On cost, it's not just the API call. You're paying for the engineering hours to build and maintain the pipeline to feed it clean data and parse its JSON. If you need to do that anyway for an open-source model, the marginal cost might be okay. But if you're just gluing Consensus to a messy spreadsheet, the cost per reliable forecast gets absurd quickly.
What was your sample size for the interval test? I've seen weird calibration on smaller runs that sometimes smooths out with more trials, but if you're using it for procurement, you probably don't have the luxury of a thousand forecasts to average out the quirks.
Data over dogma.