I've been architecting a production-grade Vault deployment on AWS ECS (Fargate) with an auto-unseal mechanism leveraging AWS KMS. The configuration operates correctly during initial deployment; Vault initializes, unseals via KMS, and functions as expected. However, upon a controlled task restart or a service recovery event—simulating a reboot scenario—the Vault container consistently fails to auto-unseal and enters a sealed state. This necessitates a manual intervention, which defeats the entire purpose of our high-availability design.
My configuration is as follows. The Vault server configuration (`vault.hcl`) includes the standard storage backend and the KMS seal:
```hcl
storage "raft" {
path = "/vault/data"
node_id = "vault_1"
}
seal "awskms" {
region = "us-east-2"
kms_key_id = "alias/vault-unseal-key"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "true"
}
api_addr = "http://${VAULT_PRIVATE_IP}:8200"
cluster_addr = "http://${VAULT_PRIVATE_IP}:8201"
disable_mlock = true
```
The IAM role attached to the ECS task has a policy granting all necessary `kms:Encrypt`, `kms:Decrypt`, and `kms:DescribeKey` permissions for the specified key. The initialization logs show a successful unseal:
```
[INFO] seal.awskms: configured
[INFO] core: security barrier initialized
[INFO] core: successfully unsealed!
```
Post-reboot, the critical error in the logs is:
```
[ERROR] seal.awskms: error decrypting data encryption key: InvalidSignatureException: Signature validation failed
[ERROR] core: failed to unseal core: error="stored unseal keys are invalid, could not decrypt"
```
I have systematically validated the following:
* The KMS key policy correctly allows the executing IAM role.
* There is no key rotation or key state change (disabled/pending deletion) in KMS.
* The region and `kms_key_id` are consistent and resolvable from within the container.
* The Raft storage path is persisted across restarts via an EFS volume.
My leading hypothesis is that the **`api_addr` and `cluster_addr`** configuration, which uses an environment variable for the private IP, might be causing a mismatch in the root token or recovery key encryption context after a restart. The AWS KMS seal uses an encryption context that includes the Vault server's configured API address. If this address differs between restarts—even if the underlying storage is the same—the decryption can fail.
Has anyone encountered and resolved this specific signature validation failure with the AWS KMS seal after a restart? I am particularly interested in:
* The precise role of `api_addr` in the KMS encryption context.
* Whether using a stable DNS name (over a dynamic IP) for `api_addr` is a hard requirement for reliable auto-unseal.
* Any potential race conditions in the ECS task definition where the environment variable for the IP is not yet populated when Vault reads its configuration.
* Alternative debugging steps beyond the standard IAM and key policy checks.
I will be conducting a test where I hardcode a stable internal DNS name for the `api_addr` and will report back with comparative metrics on success/failure rates. Any data points from similar production deployments would be invaluable.
-- elliot
Data first, decisions later.
Check your task's IAM role trust policy. On a restart, the credentials service can sometimes take a moment to be available. If the container tries to call KMS before it can obtain those temporary role credentials, it'll fail. Look at the exact error in the Vault logs just after startup. That's your real clue.
Also, using an alias like `alias/vault-unseal-key` is fine for permissions, but I've seen weird propagation delays with alias targets in some regions. Hardcode the full key ARN to rule that out.
Make sure your `api_addr` uses the actual private IP, not the placeholder variable, in the runtime config file. If it's wrong, the node can't cluster properly after a restart, which can look like an unseal problem.
IAM permissions and key ARN are valid points, but you're missing the critical order of operations. The real failure is almost certainly the environment variable expansion in your config file. `api_addr = "http://${VAULT_PRIVATE_IP}:8200"` is a dead giveaway. That variable doesn't exist in the Vault runtime environment unless you explicitly set it, and Vault parses that literal string on startup.
When the task restarts, Vault reads the config, sees `"http://${VAULT_PRIVATE_IP}:8200"` as a literal URL, and uses that for its API address. It then can't form a proper cluster or perform node-to-node communication, which leads to a cascading failure that manifests as a sealed state. The auto-unseal might technically succeed, but the node's cluster state is broken.
Replace that with the actual, concrete IP or use the instance metadata service to fetch it at runtime via an entrypoint script. Don't rely on shell-style variable expansion in a static HCL file.
Also, check your logs for errors *before* the KMS attempt. You'll likely see transport or connection errors related to that malformed address. If you post the first 20 lines of the log after a restart, I'll show you where to look.