I'm trying to move away from self-signed certificates for our internal dev services. My lead suggested using HashiCorp Vault's PKI secrets engine as an internal CA.
I've got a dev Vault instance running, and I've enabled the pki engine. I think I've got the root CA generated. But I'm a bit lost on the actual workflow for issuing certificates to, say, an internal nginx or a database.
Can someone outline the basic steps from here? Mainly:
- How do I set up a role for issuing certs?
- What's the best practice for having an app automatically request and renew a cert?
- Do I need to worry about the root certificate distribution separately?
I'm using AWS for most of my infra, so any integration tips there would be helpful too.
Good question, I'm trying to learn this myself. For the role, you can use the Vault CLI like `vault write pki/roles/my-role` with settings for allowed domains and TTL. But I got stuck on the automatic renewal part for apps.
Is the usual way to use something like cert-manager? Or do people write a small script to call the Vault API before the cert expires?
Also, you'll definitely need to get that root cert onto every machine that needs to trust the new certs. How are you planning to do that in AWS, maybe with a startup script?
Ask me in a year
You're on the right track. Since you're already in AWS, I'd recommend using their Secrets Manager or SSM Parameter Store as a middle layer for distribution, not just for the root CA but also for issuing the app certs.
For automatic renewal, a small script calling the Vault API is common for custom apps. For something like nginx, you could use a sidecar container or a scheduled Lambda that fetches a new cert via Vault's API (using its approle auth) and updates the server config. The trick is to trigger it well before the cert's TTL, maybe at 2/3 of its lifespan.
Don't forget to set up a CRL (certificate revocation list) on your PKI engine if you have services that might need their certs revoked. A lot of internal setups skip this, but it's good hygiene.
For root distribution, bake it into your base AMI or use a startup script that pulls it from a secure S3 bucket or SSM. That way you control the version centrally.
The sidecar container approach for nginx sounds interesting. Does that mean you'd run something like a small Python script alongside it that polls Vault and reloads nginx config?
I hadn't considered using SSM for the issued app certs themselves, that's a clever layer. Do you ever run into issues with secrets rotation timing, like the script updates SSM but the app hasn't picked up the new value yet?
Yeah, the root CA distribution is the real "fun" part after the initial setup. Since you're on AWS, you can bake it into your base AMI or instance user-data scripts. I've also seen teams push it out via SSM State Manager as a one-time document to install it into the OS trust store.
For the automatic renewal, I've had good luck with a lightweight Go service that uses Vault's AppRole auth. It runs as a sidecar, fetches the cert, and drops it onto a shared volume for nginx to pick up, then sends a SIGHUP. The key is making sure your Vault role's TTL is longer than your script's check interval, plus some buffer.
Also, don't forget to set up a separate intermediate CA from your root. Issue all your service certs from that intermediate. It lets you keep the root offline and safe, and if you ever need to rotate, you just generate a new intermediate without touching every machine. Saved me a huge headache last year.
The intermediate CA point is critical. I'd add that you should also use a distinct mount path for it, not just a role on the same PKI engine. Something like `/pki-int/`. Makes the security boundaries clearer and the audit logs cleaner.
For the sidecar pattern, I prefer a small cron job over a long-running service where possible. It's one less process to monitor. If you go the sidecar route, make sure it fails closed, meaning if the Vault call fails, it doesn't overwrite the existing cert with an empty file.
Build once, deploy everywhere