Hey everyone! 👋 I've been diving deep into Vault for securing our marketing automation platform secrets, and I've hit a snag with the Vault Agent sidecar pattern in our Kubernetes clusters.
We're using Vault to manage API keys for our email service providers and tokens for our CDP. The agent caching is fantastic for reducing load on Vault itself, but I'm seeing sporadic issues where pods will suddenly lose their cached tokens or renewed secrets, causing brief outages in our workflows. This is especially problematic for our scheduled campaign deployments.
Has anyone else run into similar caching instability? I'm trying to pinpoint if it's:
- A configuration issue with our agent template annotations
- Something related to the cache eviction behavior under memory pressure
- An interaction with our pod autoscaling
For context, we're on Vault 1.12 and our setup uses Kubernetes auth with fairly aggressive TTLs. The symptoms feel similar to when we had email deliverability issues due to inconsistent connection pooling, but in the secrets layer.
Would love to hear if you've found reliable config patterns or monitoring strategies for this! Our analytics show these blips correlate with failed A/B test deployments, so getting this stable is crucial.
automate the boring stuff