We are in the process of consolidating our multi-cloud observability strategy and have selected Grafana Loki as our primary log aggregation layer, primarily due to its cost-effective ingestion model and tight integration with our existing Mimir and Tempo deployment. However, we are encountering persistent and systemic query timeouts when executing moderately complex LogQL queries, particularly those involving multiple stream selectors with high cardinality labels and aggregation operators over windows exceeding 6 hours.
Our infrastructure is distributed across AWS and GCP, with Loki deployed as a microservices-style cluster (not the single-binary mode) on Kubernetes. We are using the `boltdb-shipper` index store with S3/GCS for chunks. The problem manifests specifically with queries of this nature:
```logql
sum by (namespace, pod) (
rate(
{cluster="us-east-1", namespace=~"production.*", container="app"}
|= "error"
| logfmt
| duration > 3s
| line_format "{{.msg}}"
[5m]
)
) * 60
```
This query times out consistently when scoped over `12h`, even though the cardinality in the `production` namespace is estimated to be within 10k active series for that period. The timeout is set to the default 30 seconds, and increasing it to 5 minutes merely delays the inevitable failure, suggesting a fundamental scaling or configuration issue.
We have already undertaken the following adjustments to no avail:
* Increased the `query_timeout` and `split_queries_by_interval` in the query-frontend configuration.
* Scaled up the querier component to 10 replicas with substantial CPU/memory allocations.
* Verified that our chunk retention and index sharding settings align with Loki's best practices for our volume.
The core of my architectural concern is whether we have fundamentally misaligned our expectations for Loki's query performance for analytical-style queries, or if there is a specific bottleneck in our schema (labeling strategy) or storage configuration. I am particularly interested in concrete insights on:
* The impact of the `query_range` splitting interval relative to our chunk duration.
* Whether moving to a dedicated index like `tsdb` would materially alter the performance profile for high-cardinality range aggregations.
* Any proven LogQL query patterns that avoid known performance pitfalls (e.g., excessive use of `line_format` or regex filters post `logfmt`).
Our fallback is to offload these analytical queries to a dedicated object-store query engine, but I would prefer to optimize within Loki's paradigm if possible.
Boring is beautiful
Ah, the classic "it's cost-effective until you actually query it" Loki experience. You've hit the wall everyone does when they move beyond simple tailing. That query isn't even that complex, but you're asking it to scan a 12-hour window of high-cardinality streams, parse logfmt, filter, format, and then aggregate.
The bolt-shipper index with S3/GCS is your main bottleneck here. It's cheap storage, not fast storage. Every one of those label lookups for `namespace=~"production.*"` over 12 hours is a series of object storage GET operations. The query engine has to materialize that entire index range before it even starts on the chunk data. At 10k series, that's a lot of little requests to S3, and they're all synchronous.
Have you actually looked at the query pattern's cost in S3 request charges versus just running a managed Elasticsearch cluster? I'm guessing not, because the marketing materials cleverly avoid that comparison. Try running that same query over a 1-hour window. Does it work? Probably. Now you know where the real scaling limit is.
Your k8s cluster is 40% idle.