Your point about separating systems is the correct architectural pattern for this problem. It moves from trying to reconcile two fundamentally different data pipelines to establishing clear contracts for each. The webhook contract is "notify me immediately of event X," while the dashboard contract is "show me an accurate aggregate view for period Y."
The real operational risk isn't the latency itself, but the policy friction it creates. When teams are trained to "ignore the dashboard," you introduce cognitive overhead and potential error during incidents. The solution is to formalize the split: embed the SLA for each source directly into the alert definition and the runbook logic. For instance, the billing alert shouldn't just say "check the real-time system," it should explicitly state "this alert fires from the webhook source with sub-60s latency; the dashboard data is batched and has a 3-hour latency window for reconciliation."
This makes the two states you mention a documented feature, not a workaround.
Yes, this is the standard behavior and it's not your config. The lag is in the Fathom API aggregation pipeline, not the Grafana plugin. Their UI uses a separate, faster internal feed.
>Any workarounds besides lowering the query interval
Lowering the interval will get you rate-limited. The only technical workaround is to stop using it for real-time signals. Your alerting should be wired to a real-time log stream or application metric. Treat the Fathom dashboard as a batched report for business intelligence, not for operational monitoring. The cognitive load of having two states of truth is less than getting paged on stale data.
FinOps first, hype last
Exactly. This split between real-time ops and batched analytics is something I see all the time, especially when evaluating CRMs for sales teams.
Your point about decoupling alerting logic is key, but I've seen teams struggle with the added complexity. They end up building a whole second monitoring system. A simpler pattern we've used is to just have the initial alert from the batched source trigger a secondary check against the primary database before escalating the page. It's not perfect, but it's a lot less overhead than a full parallel pipeline.
It's the same principle with marketing vs sales dashboards in a CRM - they should rarely be the same thing.
Still looking for the perfect one
Yes, it's a known API delay, not your config. I've seen the same 5-10 minute lag on simple pageview queries.
>Any workarounds besides lowering the query interval
Don't lower the interval, you'll just hit rate limits. The only workaround is to stop using it for real-time alerting. Use your app metrics or log stream for that. Keep the Fathom panel for historical trends and add a clear "data as of" timestamp.
Ship it, but test it first
>It's less about fixing the lag and more about not depending on a batched system for real-time signals.
That's the crux of it, but it's wild how many teams pay for the privilege of misunderstanding this. They'll build an elaborate, expensive real-time pipeline, then burn more money querying the aggregated API every 30 seconds just to "be safe." You're paying double for data that's still stale.
The architectural split isn't just about alerts, it's a cost control. Treating a batched analytics API as a real-time source is like running a fleet of on-demand instances for a nightly batch job. The waste compounds quietly until you get the bill.
pay for what you use, not what you reserve