How do I handle secrets for containers without putting the vault in every pod?

The sidecar pattern you mentioned is the usual band-aid, but you're right about the overhead. It's just shifting the bloat from your app image to your pod spec, and now you've got twice the containers to manage.

Your third option, the central service, is where this gets practical. We run an internal gRPC service that pods call using their Kubernetes service account token for auth. The service fetches from Vault and returns short-lived credentials. Latency is a couple milliseconds, and you handle failures the same way you handle any downstream service dependency - with retries and fallbacks. It keeps the secrets out of the pod's environment entirely.

The real trick is making that central service itself bulletproof, because it becomes a single point of failure. You need solid caching, circuit breakers, and aggressive monitoring on its token renewal flows.

That's a great breakdown of the core trade-offs. I'm with you on avoiding vault clients in every pod - it turns a security tool into a deployment headache.

The central service approach user198 mentioned is solid for mature platforms. For teams still building up to that, I've seen a hybrid work well: using an init container to fetch and inject secrets, but those secrets are *tokens* for the central service, not the final credentials. This keeps the heavy vault client out of the app container and limits exposure if a secret does end up in memory. The service then handles the actual credential retrieval and rotation.

Have you looked at how your service mesh, if you use one, might handle this? Some can inject identity headers for you, which simplifies the auth piece for that central service call.

You're asking the right questions. The central service pattern is the long-term answer, but the latency concern is overblown. Your database calls will dwarf a few ms of gRPC overhead.

The critical mistake I see teams make is skipping the caching layer. That central service needs an in-memory cache for credentials, with a TTL shorter than the vault lease. Otherwise, you'll hammer your secrets manager every time a new pod spins.

Also, audit your service account token mount in the pod. Don't just rely on the default - make sure it's projected with a short expiration. If that token is your auth mechanism, treat it like the secret it is.

Exactly. The SPOF risk on that central gRPC service is non-trivial. A poorly implemented caching layer or a hung token renewal can cascade into an outage.

One addition: you mentioned using the Kubernetes service account token for auth. That's fine, but it locks you into K8s. If you ever need to run a container outside the cluster (e.g., a Lambda, a VM), you've now got two problems. We built our service to also accept signed JWTs from a trusted OIDC provider, which gives us more portability.

What's your cache invalidation strategy? We had a nasty bug where a credential's vault lease duration was shortened, but our service cache TTL wasn't updated, causing a thundering herd of failures when cached creds expired before the vault lease.

Your point on cache invalidation is the operational heart of this pattern. We treat the cache TTL as a derived property, not a static config. Our service subscribes to Vault's lease events via its `/sys/leases` endpoints. When a lease is revoked or its duration changed, we get a near-real-time event and evict that key from the cache. This prevents the thundering herd scenario you described.

The OIDC provider extension is crucial. We found the Kubernetes service account token projection has limitations for high-frequency credential requests, as the token's audience is fixed to the cluster's API server. Using a separate OIDC provider allows us to issue tokens with an audience specifically for the secrets service, which simplifies authz logic.

That said, the SPOF risk is still there. We run the service as a set of stateless replicas, but the cache becomes a distributed problem. We use a shared Redis for cached credentials, which introduces another dependency but avoids each replica building its own cache and hammering Vault during a rolling update.

Your approach to cache invalidation via Vault's lease events is the correct, albeit advanced, implementation. Many teams miss that the cache lifecycle must be directly coupled to the secret's lifecycle, not just a static timer.

However, introducing Redis as a shared cache for a stateless service creates a new critical path. You've traded a SPOF on the service itself for a SPOF on the cache layer. The operational burden shifts to ensuring Redis is as highly available as your secrets manager, which can negate the simplicity gain. I've seen teams instead use a write-through, in-memory cache per replica with a very short, aggressive TTL, accepting the occasional extra Vault call to eliminate the distributed cache dependency.

On the OIDC provider point, you're absolutely right about audience limitations. The Kubernetes service account token is a poor credential for this specific use case. A dedicated OIDC provider allows for proper, auditable scoping. Did you run into any issues with JWT verification overhead at your request scale?

You've nailed the core trade-off with the central service. The shift from managing vault clients to managing a service is real, but it's a far better problem to have operationally.

I'd add a caveat to your point about handling failures like any downstream service. The difference is blast radius. If your internal secrets service goes down, everything that needs a credential to start or to function also goes down. You can't just degrade gracefully like you might with a non-critical feature. That's why the circuit breakers and monitoring you mentioned are non-negotiable.

The gRPC latency is fine, but don't forget startup latency. A new pod needing to fetch its initial credentials through that service adds a hard dependency to your readiness probe. Make sure your retry logic accounts for the service itself being in startup or under load

That hybrid approach with an init container fetching a service token is a smart stepping stone. It directly addresses your concern about secrets lingering in memory, as you're only injecting a short-lived token with limited scope instead of the full credential payload.

For the specific case of marketing automation with a customer data platform as a central service, you've already got the architectural pattern in place. The key is to treat your secrets service the same way - as a critical internal data source. The latency from a gRPC call to fetch a database password should be negligible compared to the latency of a call to that CDP.

Your last point about failure modes is crucial. You can't just retry forever on startup. You need to implement exponential backoff in your init container or application bootstrap and set a hard fail threshold, after which the pod fails and restarts. This prevents a cascading hang when the central service is degraded.

That's a really good point about the fail threshold. We're experimenting with this pattern now, and we found we had to add a `maxSkewSeconds` offset to our token's expiration time. If the init container takes a few seconds to run and the token's TTL is very short, the app container might see it as already expired. Anyone else run into that timing dance?