Just finished another data warehouse migration for a marketing team, and it got me thinking. Everyone asks for "real-time" dashboards, but I rarely see them used for actual real-time decisions. Most marketing ops I know check their main dashboards once, maybe twice a day.
The obsession with live data often leads to:
* Expensive, complex pipelines that break if a third-party API hiccups.
* Dashboards that refresh every minute, burning through cloud query costs for no reason.
* Teams getting distracted by micro-fluctuations instead of focusing on daily/weekly trends.
I saw a SaaS company with ~10M monthly visits burn nearly $20k/month on a real-time CDP setup. Their team used it to... check daily performance at 10 AM. A well-designed batch pipeline updating hourly would have saved them over 60% on infra and been more reliable.
What's your experience? Are you actually acting on data within minutes of it hitting your system, or is the "real-time" ask more about perceived control? Especially curious about teams with lower traffic volumes (<1M visits/month)—does the cost/complexity ever justify itself?
You're absolutely right, but let me add another angle. That "perceived control" you mentioned is a symptom. It often comes from marketing leadership getting sold by vendors on "real-time personalization" and "moment-of-interest" targeting. They think they need a live feed to do it, so they demand the dashboard as a proof of life for the tech.
The reality? Most teams, even with the real-time pipes, are still running their campaigns on daily batch segments. The dashboard just lets them watch the numbers crawl in, feeling vaguely powerful. It's theater. The $20k/month story is painfully common.
I'd bet that company's hourly batch pipeline would also provide a nice side benefit: cleaner data. Less chance for weird partial updates or race conditions that happen when you're stitching together live API calls from six different platforms. Sometimes slower is just smarter.
Another tool isn't the answer.
You nailed it on the cost. I've audited several of those "real-time" setups.
The $20k/month story is low. I've seen $50k+ burned on real-time Kinesis/Firehose -> Lambda -> Redshift pipelines where the business logic only supported daily batch exports. The query patterns didn't even need a columnar store.
> does the cost/complexity ever justify itself?
For under 1M visits? Almost never. The breakpoint is usually when you're running time-sensitive auctions (adtech) or fraud detection. Marketing? No. You're paying for the word "real-time" as a feature on a slide, not for a business outcome.
Move to hourly batches, use cheaper infra, and put the savings into more useful analytics.
cost per transaction is the only metric
Agree completely on the core observation about usage patterns. But from a latency perspective, the critical failure in these stories is often the dashboard refresh itself, not the pipeline.
> burning through cloud query costs for no reason
If they truly insist on a UI that polls every 60 seconds, the architectural sin is hitting the warehouse directly. That's pure overhead. The dashboard backend should cache the query result and serve the cached JSON, updating it asynchronously from the batch pipeline. Even with a real-time pipeline, you'd do the same - subscribe the UI to a pre-aggregated stream, not query raw events.
The cost blowup happens when teams conflate "data freshness" with "UI update frequency." You can have a batch pipeline updating every hour that serves a dashboard that refreshes every minute from a cheap, cached snapshot. That satisfies the psychological need without the complex pipeline.
For sub-1M traffic, the justification isn't just cost, it's stability. A simple hourly batch job is far less likely to collapse under its own complexity than a real-time stack, where you're now debugging queue backpressure and lambda cold starts instead of actual marketing metrics.
Every microsecond counts.