The client library's auto-renewal is the standard starting point, but for a day-long batch job, you're right to be suspicious of it. It often fails silently when the main thread is CPU-bound.
A sidecar is the "proper" way, but it's heavy. For a first pass, I'd combine the library's renewer with a pre-fetch of all secrets at startup. Store them in your process memory, not a file, to avoid the checkpoint complexity others are debating. This assumes your secrets are static for the job's duration, which is usually true for batch processing.
What language are you using? The Python `hvac` client's renewer runs in a separate thread, so if your job isn't releasing the GIL, it'll get starved. Go's client library handles it better.
pipeline all the things
Yeah, the GIL point is a real gotcha. I'm using Python and my batch job does heavy pandas/numpy stuff, which is basically all C code holding the GIL tight.
So even if I set up the renewer thread, it'll just get stuck waiting. You mentioned pre-fetching to memory, which I like. But then I'm stuck deciding on a TTL. My job could be 6 hours, could be 18 if the data volume is huge. Setting a max TTL long enough feels wasteful.
What if I pre-fetch, but also have a simple health check in the main loop? Like, every 30 minutes, try a quick, non-blocking test of the token? Not a full renew, just see if it's still valid. If it's not, then I can gracefully exit and rely on the orchestrator restart with a fresh token.
Python here too, and the GIL point is spot on. I hadn't considered that even with a renewer thread, my number-crunching could just lock it out.
The pre-fetch to memory is attractive for simplicity, but I'm worried about that max TTL assumption. What if there's a policy change mid-run that invalidates my cached token? Is that a real concern, or am I overthinking it?
You mentioned Go handles it better. Is that just because of goroutines, or is there something specific in their Vault client library?