We've been running Vault KV v2 secrets engine in production for several quarters, primarily as a configuration store for microservices. Lately, our monitoring has flagged a concerning, exponential growth in the storage backend size, far exceeding the rate of actual secret creation or update. After a deep dive, we've isolated the issue to the version metadata stored for every secret path.
As you know, KV v2 maintains a complete version history for each secret. Our problem isn't the secret data itself, but the associated metadata block for *every* version, which includes creation time, deletion time, and other internal flags. For high-churn paths—like a database password rotated every 30 minutes by an automated process—this results in thousands of metadata entries for a single path. The payload is small per entry, but the cumulative effect is massive.
Our current mount configuration is standard:
```hcl
path "sys/mounts/application" {
capabilities = [ "update" ]
allowed_parameters = {
"type" = [ "kv" ]
"options" = [ "version=2" ]
}
}
```
We are considering several approaches but are keen to learn from community experience:
* **Implementing a custom cleanup script:** Targeting paths with version counts beyond a threshold (e.g., 100) and using the `destroy` command on older versions. The concern here is operational overhead and potential impact on audit integrity.
* **Adjusting automation logic:** Modifying our credential rotation to reuse a version if the secret value hasn't changed, though this seems to defeat the purpose of versioning for audit.
* **Exploring mount options:** Are there any lesser-known tuning parameters for the KV v2 engine to auto-prune metadata after a certain number of versions or age? The documentation seems sparse here.
Key questions for the community:
* Has anyone else encountered this "metadata bloat" scenario, particularly with frequent automated rotations?
* What strategies have you employed to manage the version history lifecycle without compromising security or compliance needs?
* Are there any inherent performance degradation benchmarks for a mount with, say, 10,000 versions on a single path versus 100?
Our primary goal is to maintain a full, but reasonable, audit trail while preventing uncontrolled storage consumption. Any insights into your operational patterns or Vault's internal handling of metadata would be invaluable.
-- Ivan
Single source of truth is a myth.