Been running Cribl Stream to archive logs to S3 for a few months now, with a lifecycle rule to transition to Glacier after 30 days. On paper, it's the textbook cost-optimization move. In practice, it's been a bit of a headache, and I'm wondering if I'm the only one who's found the devil in these details.
The main issue seems to be around the S3 destination's handling of object metadata and tags post-transition. Our pipeline is set to add specific tags for cost allocation at the point of ingestion. When the objects move to Glacier, we're seeing cases where those tags aren't consistently preserved, which makes the whole "showback" part of this exercise a bit of a farce. The billing reports become a mess, splitting what should be a single cost line item.
I've also noticed some odd behavior with the multipart upload settings and how they interact with the eventual transition. Smaller objects created from a single event stream seem to transition fine, but larger, aggregated objects occasionally throw permissions errors during the transition phase, as if the S3 service principal loses the ability to act on the parts. This feels like a configuration dance between Cribl's assumed IAM role, its S3 API calls, and the bucket policies that I shouldn't have to choreograph.
Is this just a case of survivorship bias where only the people with problems are talking? Or are others running into similar gaps between the promised "set it and forget it" archival and the operational reality of making it actually work with AWS cost governance? The sales pitch always glosses over these integration specifics, but they're the whole ballgame when you're trying to justify the tooling.
Your mileage will vary