Everyone's hyping the new 'analyst' mode for its 'speed'. Tried it. It's fast, sure, until you get a subtly wrong answer.
Ran a test: same dataset, a simple daily active user count over a 90-day window with a weekly cohort filter. Analyst mode gave me a number 7% higher than my manual SQL. Reason? It was defaulting to counting distinct user IDs based on login events, not session starts. Our definition differs. The mode didn't show its work, just a pretty chart.
Speed is worthless if the metric is wrong. This is a classic black box problem. You can't audit the logic. You're trading accuracy for convenience.
When you need a real number for a report, you're still going to write the SQL. The time you saved initially you'll now spend verifying. Or worse, you won't, and make a decision on bad data.
Has anyone else done a real accuracy check, not just a demo? What was your variance?
Ask me about the cancellation process.
I'm a platform engineer at a ~150 person B2B SaaS shop, we run ClickHouse for analytics, dbt for transformations, and Metabase for internal dashboards. I've been responsible for the whole pipeline from ingestion to dashboard for two years.
Here's my breakdown on automated "analyst" modes versus manual SQL, based on implementing both:
1. **Accuracy / Trust Gap**
The variance you saw (7%) is typical. In my tests on product funnels, I've seen 5-15% differences because the tool's heuristics for "session" or "active user" never match our internal definition. One tool consistently counted bot traffic because it couldn't apply our IP exclusion list. You can't see or edit the underlying logic; you're trusting their guess.
2. **Audit Trail & Debugging**
A manual SQL query is a full audit trail. You can version it in Git, trace lineage in dbt, and peer-review the logic. With an analyst mode, you get a chart and maybe a vague "insight" description. When a stakeholder asks "why did this number spike?", you'll spend hours trying to reverse-engineer the black box, or just re-write the query from scratch to verify.
3. **Iteration Speed for Exploratory Work**
The advertised speed is real for the *first* question. If your initial vague question is "show me DAU," you'll get a chart in 10 seconds. But if you need to adjust the definition ("only web users, excluding test accounts, with a minimum session duration"), the speed advantage vanishes. You'll spend 5 minutes trying to phrase a natural language query just right, then give up and write the 4-line SQL WHERE clause in 30 seconds.
4. **Hidden Cost: Verification Overhead**
The real cost isn't the license fee; it's the team's time. Every "quick" answer from an analyst mode must be spot-checked against a known-correct manual query before it can be used for decision-making. At my last shop, this added about 20% overhead to every ad-hoc analysis request, which meant we eventually stopped using the auto-generated insights altogether.
I still use these modes, but only for one specific case: generating hypotheses during exploratory data discovery on a *new* dataset where I have no existing queries. For any metric that goes into a report, dashboard, or decision, I write the SQL myself.
If you want a clean recommendation, tell us the size of your team doing analysis and whether you have a centralized, well-defined data dictionary everyone agrees on.
Automate everything. Twice.