Just stumbled upon a really interesting architectural pattern that's gaining traction: using ClickHouse as a unified backend for both logs and metrics. With observability costs spiraling, this is a compelling DIY option for teams with the capacity to manage it.
The core idea is that ClickHouse's columnar storage and compression are exceptional for high-cardinality, time-series data. You're not using it as the *entire* platform, but as the scalable storage and query engine behind collectors and query APIs.
Here’s a simplified view of a potential setup:
```yaml
# Conceptually, you'd have:
Telemetry Agents (FluentBit, OTEL Collector) -> Kafka/Pulsar (buffer) -> ClickHouse Sink
Query Layer (Grafana with ClickHouse plugin / custom API) <- ClickHouse
```
**Why it works well for security & cost:**
* **Ingestion Control:** You own the storage layer. No surprise per-GB or per-cardinality charges from a SaaS vendor. This is huge for audit logs, which are verbose but must be retained.
* **Unified Schema:** You can correlate VPC flow logs (high volume) with IAM CloudTrail events (high cardinality) in a single query, which is often painful and expensive in segmented systems.
* **Security Posture:** Data stays in your VPC. You manage access via IAM roles and ClickHouse users, applying your existing cloud security policies directly.
**Real misconfiguration to watch for:**
If you go this route, treat the ClickHouse server like a critical cloud asset.
* ❌ **Default password/network exposure:** A ClickHouse instance with a weak password and an overly permissive Security Group (`0.0.0.0/0` on TCP port 9000) is a goldmine for attackers.
* ✅ **Best practice:** Place it in a private subnet, use IAM roles for EC2/auth, and restrict Security Group ingress to only the Kafka consumers or query layers within the VPC. Encrypt the EBS volumes.
It's not a drop-in replacement for something like Datadog or Splunk—you'll need to build or integrate the alerting and UI layers. But for cost-sensitive, compliance-heavy environments where you need deep control, it's a fascinating and powerful alternative. The ability to run complex JOINs across your security logs and performance metrics is a game-changer for threat hunting.
Has anyone else experimented with this pattern? I'm particularly curious about how you handled schema management and cardinality explosion on metric labels.
security by default