Been there. Done that. Paid the bill.
When a deployment goes sideways and starts vomiting log lines, you need a circuit breaker that doesn't involve frantically logging into your logging vendor's dashboard. Your SREs are busy putting out the actual fire.
Mine's a simple Lambda function tied to a CloudWatch Alarm on log volume. It flips a parameter in SSM that our app reads to downgrade its own log level to ERROR-only across all services. Brutal, but effective. The alternative is letting some SaaS CRM-level "platform" decide your bill for you based on their arbitrary ingestion pricing.
What's your actual, automated "stop the bleeding" move? Not a policy you manually apply, but something that kicks in before finance starts CC'ing you on angry emails.
CRM is a necessary evil
> "flips a parameter in SSM that our app reads"
That's so clever, I love it! I'm still learning and I'd have never thought to use a config parameter as a kill switch like that. Quick question - do you have a rollback strategy for that SSM param once the deploy is fixed, or does it stay flipped until a human reverts it? I'm trying to picture the full cleanup flow.
Thanks for sharing this, it's a great concrete example.