I've been evaluating the new "log boiling" feature announced by a major observability vendor this week. On paper, the premise is sound: a proprietary compression algorithm that applies lossy compression to verbose, high-cardinality log lines *before* they leave your environment, claiming to reduce egress volume by 70-90%. The vendor's benchmarks show impressive cost savings.
However, the implementation details raise immediate red flags for vendor lock-in:
* The "boiler" agent is closed-source and only compatible with their ingestion endpoint.
* The compression schema is not transparent. You cannot reconstruct the original raw log from the boiled output; you can only view it through their specific UI, which applies a "rehydration" layer.
* This creates a hard dependency. Migrating away would mean losing historical log fidelity or maintaining a parallel, expensive raw log pipeline.
Consider a basic comparison. A traditional open approach using a tool like `vector` with samplers and filters:
```toml
[sources.app_logs]
type = "file"
[transforms.sample]
type = "sample"
rate = 10
key_field = "kubernetes.pod_id"
[sinks.obsv_platform]
inputs = ["sample"]
type = "http"
uri = "https://api.example.com"
```
This gives you control over the logic and the ability to change sinks. The new "boiling" feature, in contrast, feels like moving the compression logic from a configurable pipeline stage into a proprietary black box appliance. The cost savings might be real, but they come at the expense of portability and transparency. Has anyone done a direct performance/cost comparison against open-source alternatives like columnar log pruning or head-based sampling?
benchmark or bust
benchmark or bust