I'm currently evaluating our secret management strategy for a new Kubernetes deployment, and the idea of using HashiCorp Vault as the *only* place we store and manage secrets is very appealing. Centralizing everything in one system seems like it would simplify auditing and rotation, especially for compliance.
However, I'm cautious about potential operational complexities. I've read that the initial setup and integration can be tricky. My main questions are about the day-to-day reality:
For teams running this in production, what are the less obvious pricing or operational "gotchas"? Specifically, I'm thinking about the performance overhead for pods fetching secrets at startup, and how you handle secret sprawl across multiple namespaces. Does the Vault Agent injector work smoothly at scale, or do you find yourself needing fallback local secrets for certain edge cases?
Also, from a compliance standpoint, how robust is the auditing trail when every secret request comes through Vault? Are the logs comprehensive enough for things like internal audits, or do you need to supplement them?
You've nailed the two biggest day-to-day friction points, in my experience. The performance overhead for pod startup can be significant if you're using the injector and have many init containers or a "burst" of new pods. We ended up implementing a sidecar pattern with a small, cached secret volume for our most latency-sensitive services. It adds a layer, but it was necessary.
On secret sprawl, it's a real discipline thing. Without a clear naming convention and strict path structure up front, you'll have a mess across namespaces. We treat Vault paths like a source of truth and use a GitOps flow - any new secret request requires a PR that defines its path and policies, which has helped immensely.
The audit logs are incredibly detailed, almost to a fault. For internal audits, they're perfect. For compliance frameworks like SOC2, you might need to pipe them into a SIEM or summarization tool because the volume of events can be overwhelming for an auditor to parse manually. They'll have everything they need, but you'll spend time guiding them to the right queries.
Clean data, happy life.
The performance overhead is real, especially on cold starts. If your pod startup time SLI is tight, you'll see violations. We had to move high-churn, low-latency workloads (like job queues) to using a secret cache sidecar. The injector alone couldn't cut it.
On audit logs, they're comprehensive but noisy. You'll need a separate log pipeline to filter and aggregate them for meaningful review. Raw Vault audit logs will drown your SIEM.
Secret sprawl is a process failure, not a tool one. Enforce path schemas from day one: ////. No exceptions.
Five nines? Prove it.