I've been trying to compare Power BI and Looker for a dashboard project. Every benchmark I find online seems to make one look way faster or cheaper, but the setups are totally different. Like, one uses pre-aggregated data and the other queries raw logs. Feels unfair.
How do you even do a real comparison? I'm thinking of setting up a test myself with a sample dataset on AWS. Maybe use the same RDS instance and identical queries? But I'm new to this. Has anyone done an honest side-by-side test? What should I measure besides query speed? Cost? Ease of setup?
Oh, you've peeked behind the curtain. Most of those "benchmarks" are just dressed-up marketing pamphlets. They're comparing apples to orbital satellites, on purpose.
Your test plan is on the right track, but you're missing the real pain points. Yes, use the same RDS and identical queries - that's table stakes. But you absolutely must measure the time from "I have a new business question" to "I have a chart in a dashboard." That's where the tools truly differ. It's not just query speed. It's how many clicks, how much wrestling with the semantic layer, and whether you need a degree in their proprietary modeling language.
Also, factor in the "gotcha" costs. Power BI's premium capacity fees can be a cliff, and Looker's pricing is basically a black box unless you're already negotiating with them. The setup time? Brutal for both, but in different, soul-crushing ways.
Seriously, publish your results. We need more people doing this.
Demos are just theater. Show me the real workflow.