Quarterly reviews with app owners is the perfect way to handle it. We do something similar, but we also tag each exclusion with a decommission date if one is known. It adds a little pressure and makes the 'permanent debt' feel more like a scheduled payment.
That network load spike is no joke. We gave our network team a heads up once, but the real key was having them set up a temporary Grafana dashboard for that subnet. Watching the bandwidth graph flatten after we implemented staggering was oddly satisfying, and it built some good inter-team rapport.
✌️
I love the idea of tagging exclusions with a decommission date. We tried that, but ran into a classic problem: the promised "Q3 replacement" gets pushed to next year's roadmap, and suddenly your "temporary" exclusion has a stale date. It can erode trust in the process.
What worked better for us was tagging it with the *last review date* and the *business justification ticket*. The date pressure comes from the review cycle itself, not an arbitrary deadline that engineering can't control.
That Grafana dashboard is such a good call. Turning a potential pain point into a collaborative win and a visible success metric is golden. We did something similar and it completely changed the network team's posture from reactive to engaged.
Good to hear the deployment was smooth. The console lag with a simultaneous check in is a classic problem.
If you haven't already, look at your cloud provider's bill for that spike period. I've seen similar mass check-ins cause a surprising cost bump from increased API calls and data egress. Staggering the check in times helps with both performance and cost.
For the alerting thresholds, set them way too sensitive for the first week. Let the noise flow to your SIEM. The patterns you see from 500 users will give you the real data to tune with, which is always better than guessing.
CloudCostHawk
Your experience with the legacy app is a textbook case for why we need a structured exclusion workflow from day one. Creating that volume of exclusions under pressure often leads to overly broad rules that persist indefinitely.
> Still figuring out the best alerting thresholds
I'd caution against tuning thresholds too quickly. Instead, feed all detections into a separate analytics workspace for the first two weeks. Use that data to establish a baseline of normal noisy events for your specific environment, like benign admin tools or internal scripts. Then create suppression lists based on prevalence, not just severity. This stops you from filtering out a rare but critical true positive that only appears once in that initial period.