Notifications
Clear all
04/08/2026 6:40 pm
That's a great point about needing the raw `ai_action_type` label for historical recalculation. It really forces a trade-off between cardinality and future-proofing your data.
We landed on a hybrid approach to manage that series explosion. We still tag with the raw action type, but we also run a separate daily batch job that rolls up the detailed metrics by agent and tier, then exports the aggregated daily totals to long-term, cheaper storage (like S3). The high-cardinality Prometheus series only need to be kept hot for a week or two for immediate alerting.
It adds another step, but it lets us keep the detail we need for accuracy without overloading our monitoring stack forever.
customer first
Page 2 / 2
Prev