Real-time dashboarding is the new shiny object every vendor is polishing. Spoiler: most can't do it.
They claim "sub-second latency" but that's usually on a pre-aggregated cube, not a live transactional database under load. Try pointing their "real-time" connector at a busy PostgreSQL instance and watch the dashboard grind to a halt. The polling intervals alone are a joke—often 5+ minutes unless you pay for the "enterprise" streaming add-on, which then locks you into their cloud.
Open source options like Grafana get closer, but then you're building the plumbing yourself. The big-name BI tools? They're for static reports, not live data. Their licensing models also assume you're not hitting them with constant updates. Good luck with that.
—aB
> their "real-time" connector at a busy PostgreSQL instance and watch the dashboard grind to a halt
Had this exact fight last month with one of the "modern data stack" BI tools. Their live connection just hammered our RDS with full table scans on every refresh. Support's solution? "Increase your instance size." Sure, let me just throw more money at the problem instead of fixing your garbage connector.
We ended up piping change data capture events from Postgres into Kafka, then sinking into ClickHouse for the dashboards. Grafana on top. It works, but now I'm babysitting three more moving parts at 3am. The vendor's "real time" is just a tax on your sanity.
NightOps
> Increase your instance size
Classic. When the solution to their problem is you buying more hardware, that's a broken abstraction.
Your ClickHouse pipeline is the correct move, but the operational tax is real. We run a similar pattern but use Materialize instead of ClickHouse for some views. Lets us write SQL transformations on the CDC stream, outputs to a Postgres sink that Grafana reads. Fewer pieces to babysit.
Still, it's architecture you built because the BI tool couldn't.
Trust, but verify
Your point about polling intervals being a joke is spot on, and it exposes the core architectural mismatch. These tools are built for periodic batch extracts, not continuous data flows. The 5-minute default is often the only stable configuration because their query engines can't handle a stream of updates without expensive state management.
That vendor lock-in with the "enterprise" streaming add-on is the real kicker. You're not just paying for a feature, you're committing to their entire data ingestion pipeline, which usually means duplicating your production data into their proprietary cloud store. So now your real-time dashboard is reading from a secondary, lagging copy of your data, defeating the entire purpose.
There's a middle ground you hinted at with Grafana: using a purpose-built streaming database as the buffer. Tools like Materialize or RisingWave let you define your dashboards' logic as standing SQL queries on CDC streams. Your BI tool then connects to the materialized view as a simple Postgres read replica. It shifts the plumbing burden but at least keeps the data fresh and the load off your OLTP system.