Hi everyone, I'm still getting the hang of our observability setup. My team uses Grafana with Prometheus as the data source. Lately, my dashboards have been loading super slow, sometimes taking over 30 seconds to show any graphs. This happens even for simple panels with just a few metrics.
I'm trying to understand where to start looking. Could it be the queries themselves, maybe something with Prometheus, or the Grafana instance? What are the most common beginner mistakes that cause this? Any simple checks I can do first?
Your instinct to start with the queries is correct. In my experience with Azure Monitor, even "simple" panels can trigger inefficient queries. Grafana is often just the messenger.
First, check the query inspector on a slow-loading panel. Look for the actual execution time versus just the data transfer time. A high execution time points squarely at Prometheus. Common issues I've seen:
- Unoptimized metric labels causing excessive series matching
- Using `rate()` or `increase()` over very wide windows without a suitable `resolution` step
- Querying high-cardinality metrics directly without any aggregation
Also, verify your Prometheus server's resource usage. A lack of memory can cause slow queries, as it struggles to hold time series data in memory for evaluation.
Show me the bill.