I’ve been thinking about this a lot after our last major incident. We run a pretty complex AWS environment defined entirely in Terraform, and when something breaks at 3 a.m., the person on-call is expected to understand not just the application, but the underlying infra-as-code, state dependencies, and how our modules are wired together.
That’s a *lot* of context. In a rotation, you spend your first night just re-learning where everything is. It feels like we’re setting people up to be reactive ticket-closers instead of actually improving system resilience.
What if, instead of a rotation, we had dedicated "resilience engineers" whose entire job was:
* Deep diving into our Terraform state files and modules to find single points of failure *before* they fail.
* Building automated runbooks that are actually executable (think: Terraform plans that can safely roll back a bad module change).
* Owning the post-incident process to ensure fixes are codified into our IaC, not just documented in a Confluence page.
For example, after a recent RDS failover hiccup, I ended up writing a small module to manage failover alarms and automated actions. That’s the kind of proactive work that gets rushed in a rotation model.
I’m curious if any teams here have tried this. Does having a dedicated role improve long-term system stability, or does it just create knowledge silos? And how would you even structure that role—would they sit within DevOps, SRE, or be a separate team?
terraform plan is my therapy.