Alright, let's see if we can collectively diagnose this before Claw Support sends their templated "check our docs" reply and a surprise invoice.
Situation: Using Claw for APM & custom metrics. Last Thursday, our custom metric `app.transactions.processed` (a simple counter) went from a steady 5K/minute to over 500K/minute overnight. No deployment, no traffic spike, no config change on our end. The bill projection is now comedy material.
I've ruled out the obvious:
* No duplicate metric emission from new pods (checked deployment logs).
* Not a cardinality explosion from tags—we're using a fixed set of three (`env`, `service`, `region`).
* Not a code loop gone wild (metric is emitted once per transaction, and transaction volume was flat).
Here's the kicker: the *ingested* metric count in Claw's dashboard matches the explosion, but our own internal aggregation shows the expected ~5K. So the problem is between our emission and their counting.
My suspicion points to Claw's client-side aggregation—or lack thereof. We're using their recommended Go library. Could this be a scenario where, under certain network conditions, batches are being retried and counted multiple times on their end? Their pricing doc is, of course, vague on what constitutes a "billable metric datapoint."
Has anyone else seen custom metric counts detach from reality with Claw, specifically when:
* Using histogram/summary types alongside counters?
* Having intermittent connectivity to their collector endpoints?
* After one of their "silent" agent updates?
I'm bracing for the "you're emitting wrong" response, but the numbers don't lie on our emission logs. Feels like another case of observability tools becoming the very thing they're meant to observe.
Question everything.
Question everything.