I've been knee-deep in a vendor evaluation for a new BI platform, and Grok is on the shortlist. The sales engineering team gave us the full razzle-dazzle, with particular emphasis on the "intuitive power" of their calculated field logic. They claimed it was "consistent across all visualization types" and "context-aware." Well, after a three-week proof of concept, I'm here to tell you that's a rather generous interpretation of reality.
My team has stumbled into a rabbit hole so perplexing it's making me question the entire underlying calculation engine. We have a simple, foundational metric: `Profit Margin`, defined as `(Revenue - Cost) / Revenue`. We built this as a calculated field in the data model layer, expecting it to behave like a first-class column. And it does, until you start using it in different visual contexts. A table visualization shows one aggregate number for a quarter. A bar chart, using the same dimension (quarter) and the same `Profit Margin` field, shows a different aggregate number. A line chart? A third, completely different value. We're talking discrepancies of 15-20 percentage points, not rounding errors.
The immediate assumption is that we're messing up the aggregation. So we tried forcing the aggregation at the row level before the visual aggregation, using a nested calculation: `SUM(Revenue - Cost) / SUM(Revenue)`. This *should* be the textbook, aggregation-safe way to define a ratio. In one dashboard, it works. In another, pulling from a logically identical but separate dataset connection, it yields a divide-by-zero error where none exist in the raw data. The behavior is inconsistent across dashboards, even within the same instance.
I've engaged their support, and the responses are a masterclass in obfuscation. They talk about "calculation scopes," "implicit filters," and "evaluation contexts," but refuse to acknowledge that the engine is applying logic inconsistently. They keep asking for our dataset schema, as if the problem is our data cleanliness and not their black-box arithmetic. It feels like the calculation is being evaluated *after* visual filters and aggregations in some contexts and *before* in others, with no clear indicator or user control.
Has anyone else performed a serious, non-toy evaluation and hit this wall? I'm not interested in "did you check your joins" boilerplate. I'm looking for concrete evidence that this is a known architectural flaw in Grok's calculation engine, or better yet, a documented (but well-hidden) setting that actually locks down calculation behavior. The security and compliance implications of reporting inconsistent financial metrics from the same system are, to put it mildly, significant.
Just my 2 cents