I'm running a mid-sized deployment (~50 services, 20 devs) and our observability bill is getting stupid. We're ingesting about 100GB of application logs per month, plus some metrics and traces. Commercial platforms want $1000+ for that. I'm drawing a hard line at $500/month, ideally on a cloud VM, and it needs to be open-source core.
I've tested a few combos, but everyone's marketing is full of "scale to infinity" nonsense. I want real-world, production-tested opinions.
My current front-runner is:
* **Loki** for logs (object storage backend)
* **Prometheus** + **VictoriaMetrics** for metrics (single-node VM)
* **Tempo** or **Jaeger** for traces
* **Grafana** for dashboards/alerts
The ingestion pipeline is the tricky part. Do you use:
* Grafana Agent (all-in-one, but can be a SPOF)
* Vector or FluentBit (more flexible, another component to manage)
* Direct writes from Otel collectors?
Biggest concerns:
1. **Query performance** on 100GB/day in Loki. Do I need to over-provision the queriers/index gateways?
2. **Cardinality explosion** in VictoriaMetrics. Are we talking about needing `vmalert` and strict labeling from day one?
3. **Maintenance overhead.** What's the actual time cost per week to keep this running smoothly?
Here's a snippet of a Grafana Agent config I'm testing for a Kubernetes setup:
```yaml
metrics:
wal_directory: /tmp/grafana-agent-wal
global:
scrape_interval: 60s
configs:
- name: kubernetes
remote_write:
- url: http://victoria-metrics:8428/api/v1/write
kubernetes_sd_configs:
- role: pod
logs:
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
configs:
- name: applications
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
# ... standard relabeling to get pod logs ...
```
What's your actual stack? Where did you hit walls? If you had $500 and 100GB/day, what would you deploy tomorrow?
Show me the latency.