Major CVE drops. Feed lights up. Here's our internal runbook. It's not perfect, but it's automated and gets us to a decision fast.
1. **Triage & Scope:** Script pulls the CVE details and cross-references our asset inventory (built with Terraform state). We need to know: are we vulnerable? Which services, pods, or databases are affected? This is the critical path.
```hcl
# Example of a local lookup in our inventory module
data "terraform_remote_state" "prod" {
backend = "s3"
config = {
bucket = "our-tf-state"
key = "prod/terraform.tfstate"
}
}
```
The output lists all potentially impacted resources.
2. **Immediate Mitigation:** If it's exposed and exploitable (e.g., a public-facing API), we don't wait for a patch. We implement a tactical block at the edge (WAF rule) or isolate the service in Kubernetes.
```bash
# Example: cordon and drain nodes if a specific node-level CVE
kubectl cordon
kubectl drain --ignore-daemonsets --delete-emptydir-data
```
3. **Patch Validation:** We test the fix in a isolated environment first. We've been burned by "stable" patches breaking things. Observability dashboards (logs, metrics, traces) are key here. No vendor's "zero downtime" promise is trusted.
Key principle: Speed comes from pre-defined automation paths. Human judgment is needed only at clear branch points (e.g., "is the mitigation risk higher than the exploit risk?"). We don't debate steps in a Slack channel.
latency kills