Made the switch last quarter. Primary metric: trace query latency. Secondary: ingestion cost.
Honeycomb results:
* Query latency: 1.2s avg (Lightstep was 2.5s+)
* Ingestion cost: $3.2k/month (Lightstep was ~$1k)
* Dataset: ~2TB ingested monthly, high cardinality attributes.
Key config difference: Honeycomb's derived columns are powerful but expensive. Lightstep's sampling was more aggressive by default.
```json
// Honeycomb derived column for error rates
{
"name": "error_rate",
"expression": "BOOL($http.status_code >= 500)",
"type": "BOOL"
}
```
Query speed is transformative for debugging. Cost is a major hit. Running a break-even analysis now.
Question: Anyone else hitting high costs with high cardinality on Honeycomb? What's your workaround besides dropping attributes?
Benchmarks don't lie.
Senior FinOps lead at a 500-person SaaS company. We ingest ~5TB of observability data daily, split between Datadog, Honeycomb, and in-house logging. Run Honeycomb exclusively for microservice debugging.
**Key cost drivers you're hitting:**
* **Derived column pricing:** This is your primary leak. Each derived column is a separate, billable data stream. That `BOOL()` operation on high-cardinality data is likely reprocessing your entire ingestion volume. We saw a 40% cost increase from three derived columns.
* **High cardinality tax:** Honeycomb's model is built for cardinality, but you pay for it. Costs scale directly with unique value combinations. At 2TB/month, you're in the territory where a 10% reduction in cardinality can cut your bill 15-20%.
* **Sampling discipline:** Lightstep's aggressive sampling is a cost control. Honeycomb's default sampling is minimal to preserve fidelity. You must implement sampling before ingestion (e.g., at the OpenTelemetry collector) or use Honeycomb's `sample_rate` attribute.
* **Query speed vs. storage:** The inverted index that gives you 1.2s queries is expensive to maintain. Lightstep uses cheaper, slower object storage for some data. You're paying for that index on every high-cardinality attribute.
**Workarounds we enforce:**
1. Move derived logic upstream into instrumentation where possible. Calculate `error_rate` in your app code, send as an attribute.
2. Implement deterministic sampling at the collector. Start with a 50% sample on non-error traces.
```yaml
# OTel Collector processor config
probabilistic_sampler:
sampling_percentage: 50
hash_seed: 22
```
3. Use Honeycomb's `DROP` rule to discard high-cardinality debug attributes you never query.
4. Negotiate a committed use contract. At your volume, you should get a 20-30% discount off list.
**Pick:** Stick with Honeycomb for now. The query speed directly translates to engineer productivity. Your break-even is whether the $2.6k/month increase saves more than ~10 engineer hours per month in debugging time. If cost is still prohibitive, you need to tell us your acceptable query latency threshold and whether you can impose strict schema discipline on dev teams.
cost per transaction is the only metric