We're hitting the wall with our current setup when analysts run ad-hoc queries on our 10TB+ ClickHouse data lake. The dashboard tool just bogs down.
I'm testing tools that can handle this scale natively, without needing to pre-aggregate everything into a cube. Looking at engines like Apache Superset with a direct query layer, or specialized platforms like ClickHouse's own dashboarding.
Anyone running similar loads? What's your stack? Specifically interested in:
- Real-world query performance on massive joins
- How they handle 100+ concurrent users
- The total cost of scaling to this level
I'll share my prototype results next week. ->
Automate everything.
I'm a principal engineer at a fintech processing ~15TB of transactional data, and we serve real-time dashboards directly from our ClickHouse cluster using a mix of tools.
* **Concurrency & Connection Management**: Metabase's direct query to ClickHouse can become the bottleneck itself. With 100+ concurrent users, the single PostgreSQL backend Metabase uses for query coordination becomes overloaded, causing queueing. We saw 3-4 second delays on query submission at ~70 concurrent users. Tools that push all logic to the database handle this better.
* **Cost Scaling for User Load**: The per-user pricing of cloud services like Looker explodes at your scale. At 100+ users, you're easily into $40k+/year. Open-source options like Apache Superset have no user-based cost, but their scaling cost is your engineering time to manage and optimize the coordinator nodes.
* **Performance on Large Joins**: This depends almost entirely on your database, not the BI layer. However, the tool's query generation matters. We found Superset's default SQL for complex joins could be inefficient, requiring manual `SQL Lab` edits. A tool that generates a clean `WHERE IN` clause can be 10x faster than one that materializes a massive subquery.
* **Hidden Deployment Cost**: The biggest hidden cost is the "pre-aggregation tax." If a tool forces you to build cubes because its direct query is poor, your data team becomes ETL engineers. We tested one platform where our POC required 50% of our raw tables to be pre-computed to be usable, adding $12k/month in new pipeline costs.
For your stated need of ad-hoc queries on 10TB+ without pre-aggregation, I'd recommend a well-configured Apache Superset deployed close to your ClickHouse cluster, but only if you have the engineering bandwidth to manage it. If you lack that, I'd look at ClickHouse's own Grafana integration for a simpler path.
Tell us about your team's size and whether your analysts are comfortable writing SQL directly, and I can narrow it down.
sub-100ms or bust
Interesting thread! The scale you're working at is way beyond my current experience, but I'm really curious about your results.
How do you even begin to test concurrency at that level? Are you simulating 100+ users, or is that your actual team size already hitting the system?
I'm looking at much smaller datasets for my team's project management metrics, but performance issues already pop up. I can't imagine joins on 10TB. Good luck with the prototypes, I'll be following this.