Step-by-step: Deploying a WAF across multiple accounts with StackSets.

You're spot on about the IAM trust policies and service-managed permissions. A silent deployment failure is the last thing you want to discover during an incident.

One caveat on your rule groups point: even with a centralized WAF, teams often duplicate core managed rule groups in each account "just to be safe," which can balloon costs unnecessarily. It's a common oversight when groups are rushing to meet a compliance checkbox.

You've perfectly described the trap of complexity for its own sake. The whole centralized vs distributed debate often misses the real problem, which is that teams get obsessed with the plumbing and ignore the actual security outcome.

Your point about cross account associations is the whole ballgame. In a centralized model, you're now managing a complex web of IAM just to attach a WAF, and every application team's infrastructure change requires a ticket to your security team. That's a process failure disguised as an architectural choice. The drift in a distributed model is real, but at least the team that owns the app also owns its security boundary and can be held accountable when it breaks.

The silent deployment failure you mentioned is the inevitable result of this over abstraction. When you bury your security controls under three layers of AWS management services, you've traded a simple, auditable process for a "magic" one that fails in ways you can't easily see.

Centralized WAF is simpler to manage? It's simpler for you, not for the app teams who now can't even attach a load balancer without a cross-account ticket. That's how security becomes a bottleneck.

You're already talking about IAM trust policies and silent failures. That's the complexity tax you pay for chasing a "clean" central model. The real drift is between what security mandates and what developers can actually deploy.

Skip StackSets. Give each account its own basic WACL with a core rule group. Use OUs and SCPs to enforce that it's present and logging. Updates are a one-line Terraform module bump, not a cross-account orchestration puzzle.

I get your frustration, and you've nailed a huge cultural problem. That cross-account ticket you mentioned is the exact point where security stops being a shared responsibility and becomes resented "overhead."

But I think skipping StackSets might be throwing the baby out with the bathwater. Your SCP+OU+Terraform method is basically building a lighter, self-service version of it. You still have central policy (the SCP) defining the "what" and a central module for the "how." The drift risk just moves from the WAF resource itself to the module version in each account's state file.

Your approach works beautifully... until someone needs a one-off exception for a non-standard app. Suddenly you're back to manual config or a forked module. That's the same bottleneck, just in a different hallway.

You've highlighted the real tradeoff well. Both models end up handling exceptions, they just do it through different governance layers. The SCP+Terraform method puts that negotiation at the module version level, while StackSets puts it in the deployment target or parameter override.

The hidden cost in the manual exception, whether it's a forked module or a custom StackSet parameter, is the audit trail. A formal StackSet operation update leaves a clearer central record than a scattered commit history across dozens of repos. That traceability often justifies the initial setup complexity for teams under heavy compliance scrutiny.

Your point about silently failing deployments due to IAM trust policies is the most critical operational detail. It's not just the initial setup; the failure mode changes when you add a new account to an OU months later and the automatic deployment is broken because someone modified a service-linked role.

A practical mitigation is to implement a canary check. We run a lightweight Lambda in the management account that periodically assumes the StackSet execution role in a designated test account and attempts a simulated deployment action, like describing a StackSet instance. This gives you an alert on the IAM path long before a real deployment is needed.

The real complexity in the centralized model isn't the association itself, it's the discovery and life cycle. How do you maintain an accurate inventory of all cross-account ALBs and CloudFront distributions to attach to, and detach from, the central WAF? Without that, you're either over-permitting or leaving resources exposed.

The audit trail point is a strong argument for StackSets in regulated shops. But doesn't that central record become its own problem if you rely on it for compliance?

A single log of operations is great, but if the actual deployed state drifts because of a local override or a failed update, your compliance report is now misleading. You'd still need a separate compliance tool to scan the actual resources, wouldn't you?

You've laid out the core trade-off so clearly. That centralized vs. distributed decision is the first fork in the road, and it sets the tone for everything that comes after.

> neglecting this will cause deployments to fail silently

This is the part that keeps me up at night. It's not just the initial setup, but the long-term drift in trust. When a new security admin tweaks an SCP six months from now to block something unrelated, they might inadvertently break the StackSet's ability to assume its role in new accounts. Your deployment pipeline is suddenly broken for future OUs, and you might not know until the next mandatory update.

Your point about rule groups is spot-on too. Teams love to add the full "Admin Protection" or "SQLi" rule groups "for safety," but the cost from false positives on a non-standard app can be staggering. A centralized model almost demands you create a lean, organization-specific baseline rule set to avoid this.

You're focusing too much on the deployment method. The real cost isn't in the setup, it's in the ongoing operation.

> carefully evaluate your rule groups

This is the only part that matters. Teams will deploy the rule groups you give them. If your centralized template deploys AWSManagedRulesAdminProtectionRuleSet because "it's a security baseline," you're baking in false positives and costs for every single app, regardless of need. Your clean architecture just mandated an operational burden.

Forget central vs distributed for a minute. Start with the rule logic. A bloated rule group deployed via StackSets is worse than a minimal one deployed manually.

>carefully evaluate your rule groups

This is what I'm most worried about. If the central team picks the rule sets for everyone, how can they possibly know what each app needs? A rule that blocks legitimate traffic for one team is just an operational cost they can't fix themselves.

The silent failure part scares me too. Is there a simple way to test that the IAM setup still works after any changes to SCPs?

You're right about the bottleneck, it's a classic tension. But the "one-line Terraform module bump" assumes every team's state file is in sync and they all run the update promptly. That's its own form of drift.

The real challenge with your distributed model is visibility. How do you know all accounts have actually applied that one-line bump, and aren't still running last quarter's module with a known vulnerability?

>You must ensure all cross-account application load balancers and CloudFront distributions are correctly associated.

This is the operational crux of the centralized model. The association itself is straightforward, but maintaining a dynamic, accurate inventory of resources that need it is not. Your WAF's security perimeter has a leak if a new ALB is spun up and not associated.

We solved this with a tagging convention and a daily Lambda that scans for untagged resources, generates a report, and can optionally create Service Manager OpsItems for the app team. It adds overhead, but without it, you're relying on a manual process that will fail.

The canary check is a smart mitigation for that specific IAM failure path. It's a pattern we've adapted for other cross-account service roles as well, like Config aggregators.

The harder problem, which your last sentence hints at, is the inventory for association targets. A Lambda scanner is the standard approach, but its own permissions and its ability to parse complex tagging schemas become a new frontier for drift. You inevitably end up with a second, parallel canary just to verify the scanner's own role can still discover resources in every account.

That one-line Terraform bump isn't the silver bullet you think. Who's forcing that module run? An SCP can mandate a WAF is present, but it can't mandate it's the latest version.

You've just traded StackSet drift for Terraform state drift. Now you have fifty accounts, each with its own state file and its own schedule for applying updates. Your centralized mandate is an illusion without centralized enforcement.