Hey everyone! As our team grows and our on-call rotation gets more complex, I've been thinking a lot about how to keep our incident response skills sharp. We've got great monitoring with Prometheus and Alertmanager, and our PagerDuty setup is solid for real alerts. But running a real fire drill always feels risky—nobody wants to accidentally wake up the entire engineering team at 2 AM because a training exercise went sideways! 😅
I'm curious: what strategies or tools have you all used to simulate realistic incidents for training purposes, without those simulations bleeding into production alerting? I'm picturing scenarios like:
* Simulating a database latency spike
* A mock deployment failure in a staging Kubernetes cluster
* A "what if" scenario where a core service dependency goes down
Ideally, I'd love a solution that lets our on-call engineers practice triage, communication, and runbook execution in a safe, controlled environment. Do you create a separate, mirrored alerting pipeline for a staging namespace? Use feature flags in your alerting rules? Or perhaps there's a specific tool built for this kind of non-production incident simulation?
For example, in our Alertmanager config, I've wondered about using a special `training` receiver route, but ensuring it's *never* used by mistake feels tricky.
```yaml
routes:
- match:
severity: 'training'
receiver: 'training-team'
continue: false # Make sure training alerts stop here
```
How do you handle the separation of concerns? Any gotchas or success stories to share?
Keep deploying!
Keep deploying!