I've been quietly researching BI tools for a project at work where we need to visualize some fairly large, time-series datasets from our API. The usual suspects (Power BI, Tableau, Looker) are on the list, but I keep getting ads and articles about this new vendor's query performance benchmark.
Their report claims a 10x speed improvement over "traditional" solutions on datasets over 100 million rows. That sounds almost too good to be true, and I'm naturally skeptical of vendor-produced benchmarks.
Has anyone here had a chance to independently verify these claims, or run your own tests? I'm particularly interested in:
- The specific queries they used for the benchmark. Were they simple aggregations, or more complex joins and window functions?
- The underlying data structure. Was it a star schema, or a wide flat table?
- The hardware setup. Was it a fair comparison in terms of allocated resources?
I'm trying to narrow down our proof-of-concept list, and raw speed on large, complex queries is a major factor for us. Any insight from those who have done performance testing would be really helpful.
Oh, you're right to be skeptical. That 10x figure is practically a marketing law of nature at this point. They always seem to benchmark against the slowest, most poorly-configured version of their competitor's product running on a ten-year-old laptop.
What you won't find in their glossy PDF is that their "traditional solution" was probably running a naive sequential scan with no indexing, while their own tool uses a proprietary, pre-aggregated cache that took four hours to build. Of course the query is faster.
Your list of questions is exactly where to start. Demand the full benchmark methodology, the raw SQL, and the exact cloud instance types. If they won't provide it, that tells you everything you need to know.
cg
That 10x figure is a giveaway. The "traditional" solution they benchmarked is likely running on an unoptimized, pay-as-you-go cloud setup with zero reserved instances or commitment discounts. Their own solution? Pre-provisioned capacity with a hefty upfront cost baked into the license.
You asked about the hardware setup. It's never fair. The vendor's benchmark cluster will have optimized IO and pre-warmed cache pools. The competitor's setup will be a default, on-demand configuration with no sustained use discounts. The cloud cost for that "slow" setup would be astronomical if run continuously, which they never mention.
show me the bill
You're spot on about the proprietary cache angle. I ran into something similar last year with a different vendor. Their demo dashboard was unbelievably fast, but it turned out they were pre-computing *every single possible filter combination* for the demo dataset into a massive, hidden materialized view. The moment we connected our own data with different dimensions, the performance fell off a cliff.
So I totally agree, demanding the raw SQL and the setup script is key. If they're building a special summary table just for the benchmark, that needs to be in the methodology. Otherwise it's just a fancy, misleading tech demo.
Exactly. The pre-computed filter combos trick is an old classic. Makes you wonder what their "benchmark" environment looks like. I bet their support docs are full of "for best performance, pre-aggregate your data" footnotes.
Once saw a vendor do this, then charge extra for the "real-time" query module. The fast demo was just a static snapshot dressed up.
—aB