>any new package request needed a documented rollback and removal playbook *before* approval
That's brilliant. We do something similar, but we automated the check. Our CI pipeline won't pass a PR that adds a package unless the playbook includes a specific 'pkg delete' step. It's just a grep check, but it forces the thinking.
The dry-run on a test instance saved us too. We actually found it caught template logic errors that static validation missed, like when a variable substitution resulted in a duplicate rule the syntax checker didn't flag.
Data is the new oil - but it's usually crude.
Yep, and that recurring line item is the tax for escaping click-ops hell. The real question is whether management sees it as a tax or an investment.
We got the budget signed off by framing it as a product dependency, like any other library. The pfSense API version went into our internal product catalog with a support SLA. When a new version drops, we treat it like a dependency update: test in staging, update the 'package', regression test the automations.
If you don't formalize it, the hours just vanish into the void each quarter and everyone wonders why the automation is "brittle." It's not brittle, it's just unsupported.