Datadog's 4-hour outage last week took down metrics, logs, and APM. Their own status page was delayed. If your monitoring, tracing, and alerting are all in one system, you're blind when it fails.
This isn't just about resilience. It's about cost and lock-in. Their pricing model encourages you to ingest everything, but querying that data later is expensive. For high-cardinality use cases (like per-customer dashboards), costs explode.
Questions for the thread:
* What are you using to decouple ingestion from storage/querying?
* Are you running a dual-vendor setup (e.g., one for logs, one for metrics)? What's the operational overhead?
* Any concrete benchmarks on query latency/cost for high-cardinality data between platforms like Honeycomb, Grafana Cloud, Sentry, or self-managed M3DB/VictoriaMetrics?
Example of a simple dual-write setup we're testing to buffer against vendor outages:
```yaml
# OTEL collector config snippet
exporters:
debug: {}
datadog:
api:
site: datadoghq.com
key: ${env:DATADOG_API_KEY}
otlp/grafana:
endpoint: "otlp-gateway.grafana.net:443"
headers:
authorization: Basic ${env:GRAFANA_PROM_OTLP_AUTH}
service:
pipelines:
metrics:
receivers: [otlp]
exporters: [debug, datadog, otlp/grafana]
```
Totally feel that blindspot problem. We got burned by a similar Grafana Cloud incident last year and started splitting things up.
On the high-cardinality cost explosion you mentioned - that's what pushed us to separate storage. We're using the OTEL collector to fan out to Grafana Cloud for metrics (their Prometheus pricing is better for our query patterns) and to a self-hosted Loki cluster for logs. The overhead isn't zero, but it's less than I feared, maybe 10-15% more config management.
Has your team looked at ClickHouse for log storage? The query performance for high-cardinality data is pretty wild, and it decouples completely from the vendor's query engine.
Always testing.