I've been architecting a multi-tenant Vault deployment for a regulated environment where comprehensive audit logging is non-negotiable. We enabled the `file` audit device with JSON formatting, as it's the default recommendation for most integrations. After a few weeks of operation, the sheer volume of logs became a critical infrastructure issue. The primary Vault instance, which handles a significant authentication and secret-generation load, began generating over 450 GB of audit log data *per day*. This quickly overwhelmed the local node's ephemeral disk and threatened to impact the stability of the Vault service itself.
The core issue, as I analyzed it, stems from two primary factors:
* **The granularity of Vault's audit log:** Every authenticated request, including token renewals, secret reads (even for static credentials), and configuration changes, generates a full log entry. For high-traffic services using dynamic secrets, this is multiplicative.
* **The verbosity of each entry:** A single log entry contains the full request and response payloads, including the complete data block of a `kv-v2` read, which can be large if you're storing certificates or other bulk data. The response for a successful `database/creds` read includes the generated username and password.
A naive solution would be to simply increase disk space, but that merely postpones the problem and increases cost. I've been exploring a more architectural approach. Here is a snippet of the Vault configuration we're testing to mitigate this, focusing on the audit device options:
```hcl
# In your Vault server configuration (e.g., config.hcl)
audit "file" {
type = "file"
path = "/vault/logs/audit.log"
format = "json"
# Critical parameters for volume control
mode = "744"
hmac_accessor = false # Reduces log size by omitting the HMAC'd accessor per entry
log_raw = false # DO NOT set to true in production unless absolutely required.
# It logs sensitive data like passwords in plain text.
}
```
Furthermore, we've implemented a pipeline to offload and manage these logs:
1. **Immediate offloading:** The audit log file is streamed via a Fluent Bit sidecar container (in our Kubernetes deployment) to a centralized log aggregation system (e.g., Elasticsearch), with strict retention policies.
2. **Local retention:** A logrotate policy aggressively rotates and compresses the local file, keeping only a few hours of data on the Vault node itself.
3. **Content filtering:** We are evaluating the `sink` stanza in the audit device configuration to redact specific response data keys, though this requires careful testing to avoid breaking compliance requirements.
My questions to the community are:
* Have you encountered similar scale issues, and what was your long-term architectural strategy? Did you move to the socket audit device for better integration?
* How do you balance the need for comprehensive audit trails (where every secret access *must* be logged) against the practicalities of storage and analysis?
* For those using Vault's PKI or secrets engines with large payloads, have you found effective ways to redact or omit the bulk data from the audit logs while still maintaining a valid audit trail?
The operational burden of managing these logs is non-trivial, and I'm concerned that without careful design, the audit system itself can become a single point of failure and a major cost center.
450GB/day is a lot, but it's not surprising given the default `file` audit device. That device is synchronous and blocks Vault's request pipeline until the write is confirmed. Under high throughput, it becomes a bottleneck and a disk I/O problem. I've seen teams hit this exact wall.
A few things to consider:
* **Switch to `syslog` or `socket` audit device.** The `file` device is convenient for dev but borderline irresponsible for production at scale. Both `syslog` and `socket` push the I/O off the critical path, and you can point them at a local syslog daemon that handles rotation, compression, and forwarding. `socket` with a Unix domain socket and a local syslog-ng or rsyslog instance is a common pattern.
* **Audit device options.** You can reduce payload size by setting `log_requests = false` if you only need responses, or vice versa. For regulated environments where you need both, consider using `hmac_accessor = true` to shrink the sensitive fields. Vault also supports `elide_list_responses = true` for list operations, which can cut down on massive `kv-v2` list responses.
* **Don't store audit logs on the Vault node's disk at all.** Use a sidecar process (fluentd, vector, or even a simple syslog forwarder) that reads from a local socket, compresses, and ships to S3/GCS/Blob Storage. The Vault node should only hold a small buffer. This also makes your log pipeline independent of Vault's health.
The real trap is that the `file` device is the default in the docs and tutorials, so people assume it's production-hardened. It's not. Fix your audit device, then tune the verbosity, and you'll cut the volume by an order of magnitude without losing the audit trail.
Boring is beautiful