Having recently completed a quarterly infrastructure audit, I was tasked with rationalizing our log observability spend, which had grown unchecked across several teams. The primary candidates were our existing managed ELK stack (Elastic Cloud), a Papertrail trial for some legacy applications, and a proof-of-concept deployment of self-hosted Grafana Loki. The marketing claims around "cost-effective" and "high-performance" logging are, as usual, wildly divergent from measurable reality. I structured a two-week load test to quantify ingestion throughput, query performance, and operational overhead under a controlled, reproducible workload.
The test harness simulated a production-like environment generating structured JSON application logs (Nginx access, app errors, and audit events). Key metrics were cardinality (controlled via unique user IDs and request paths), volume (scaled from 50 GB/day to 250 GB/day), and query patterns (simple filtering, complex aggregations, and full-text search). All systems were provisioned with roughly equivalent resources (8 vCPU, 32 GB RAM, 500 GB NVMe storage) in the same cloud region, with the managed services using their "medium" tier plans. The Loki stack used the `microservices` mode on Kubernetes with `boltdb-shipper` for index storage in S3.
**Summary of Key Findings:**
* **Ingestion Cost & Throughput:**
* **Loki** demonstrated the lowest raw storage cost by a significant margin, owing to its chunk-based compression and decoupling of index from log data. However, ingestion throughput per unit of compute was lower, requiring more aggressive horizontal pod autoscaling of ingesters to handle spikes, which added complexity.
* **Papertrail** was the most expensive at scale on a per-gigabyte basis, but its ingestion pipeline was remarkably consistent and "fire-and-forget," with zero operational overhead.
* **Managed ELK** presented a paradox: its compute-heavy indexing made ingestion costly at high cardinality, but its raw ingestion throughput was the highest and most consistent once resources were provisioned.
* **Query Latency (95th percentile):**
* For simple tailing and filtering by label (`{app="api", level="error"}`), **Loki** was sub-second, as expected.
* For full-text search across a 24-hour window, **Managed ELK** was 2-3x faster than Loki's LogQL equivalent.
* **Papertrail** queries were consistently slowest for complex searches, though adequate for live tailing and basic filtering.
* **High-Cardinality Impact:**
* The managed ELK cluster's performance degraded linearly as we introduced high-cardinality labels (adding a unique `session_id` to every log line). CPU consumption on the ingest nodes increased by over 300%.
* Loki's performance, when labels were carefully constrained, was unaffected by high cardinality within the log stream itself, as it only indexes the labels.
* Papertrail does not expose fine-grained control over indexing, making cardinality costs opaque but seemingly absorbed into their pricing tier.
* **Operational & Alerting Reliability:**
* **Loki's** alerting (via Grafana) was reliable but required a separate Prometheus/Alertmanager deployment for a full solution, adding moving parts.
* **Managed ELK's** integrated alerting was the most feature-complete and reliable, with native support for complex conditions on aggregated data.
* **Papertrail's** alerting was basic but 100% reliable for simple string-matching thresholds.
**Configuration Snippet for Loki's Scalability Tuning:**
The critical factor for Loki was tuning the `ingester` configuration to balance memory usage and flush frequency to object storage. An under-configured setup led to query failures due to incomplete chunk flushing.
```yaml
ingester:
lifecycler:
ring:
replication_factor: 3
chunk_idle_period: 15m
chunk_block_size: 262144
max_chunk_age: 1h
target_chunk_size: 1572864
```
**Bottom-Line Cost Model (Projected for 1 TB/day, 30-day retention):**
* **Self-hosted Loki:** ~$1,200/month (primarily compute for ingesters/queriers + S3 storage)
* **Managed ELK (Hot-Warm):** ~$8,500/month (Elastic Cloud list price)
* **Papertrail:** ~$11,000/month (based on volume pricing)
The choice is not trivial. If your team has the Kubernetes expertise and your query patterns are label-centric, Loki's cost savings are compelling despite its steeper learning curve. If you require rich, ad-hoc full-text exploration and have a budget to match, a managed ELK stack is superior. Papertrail serves a niche for teams needing absolute operational simplicity for basic log aggregation, but its cost at scale is prohibitive.
I am interested in critiques of this methodology, particularly regarding long-term index management for Loki and alternative scaling configurations for Elasticsearch that might bend the cost curve.
-ek
Show me the numbers, not the roadmap.