I've been working on a data platform where we use Terraform to manage our BigQuery datasets, Snowflake warehouses, and even some of our dbt project scaffolding. The missing piece has been secret management for service accounts and database credentials.
We're considering integrating HashiCorp Vault into our CI/CD pipeline (GitHub Actions) to dynamically fetch secrets during deployment, rather than storing them as environment variables or using Terraform's state for sensitive values. The theory sounds solid, but I'm curious about day-to-day realities.
* How are you handling authentication for the CI/CD runner to Vault itself? Short-lived tokens, JWT auth with GitHub, or something else?
* Have you run into issues with Terraform providers needing secrets at plan time versus apply time? This seems like a potential friction point.
* What does your workflow look like for rotating a secret that's already been provisioned? Does it require a full terraform apply, or do you handle it outside the IaC loop?
I'm particularly interested in experiences where data pipelines are involved—like ensuring a Airflow or Dagster deployment can access fresh credentials from Vault without embedding them in DAG code. Any gotchas around networking, performance, or secret leasing would be great to hear.
We use the GitHub JWT auth method for our runners. It works well, the token is automatically short-lived. The friction point you mentioned about plan vs apply is real. Some providers need secrets at plan time, which can be a blocker if those secrets aren't in the state but are fetched dynamically. We had to restructure a few modules because of it.
For rotation, we handle it outside the IaC loop most of the time. If a database password changes in Vault, we don't want Terraform to force-recreate the database user. We use Vault's dynamic secrets for that, so the actual credential rotates automatically and the Terraform-managed user stays the same.
How do you handle the plan-time secret issue with BigQuery or Snowflake? Do you store service account keys in Vault and pull them in during the run?
You're asking the right questions, because the gap between theory and reality here is a canyon. The JWT auth works fine technically, but it's that plan-time secret issue that becomes a massive architectural headache.
> ensuring a Airflow or Dagster deployment can access fresh credentials from Vault without embedding
This is where you have to divorce your IaC from your runtime secret delivery entirely. Trying to make Terraform provision resources *and* be the conduit for runtime secrets is a trap. We provision the service account or database user with a placeholder in Terraform, but the actual credential is a dynamic secret from Vault that Airflow fetches on boot. Terraform never sees it. If you try to feed runtime secrets through Terraform outputs, you're just recreating the problem you wanted to solve, with more steps.
And rotation? If your secret is in Terraform state, you're already doing it wrong. A full apply for a rotated password is a guarantee you'll eventually break something at 2 AM. Vault's dynamic secrets or a separate rotation process outside Terraform is the only sane path.
Spot on about divorcing IaC from runtime secrets. That's been the key for us too.
One nuance: we still use Terraform to manage the Vault policies and roles themselves. So while Terraform never sees a runtime secret, it does define *who* (like our Airflow instance) can read which dynamic secret path. That keeps the permission model codified.
The 2 AM applies for password changes hit home. We got burned once before switching to Vault's dynamic database credentials. Now rotation is just Vault's problem, and our pipeline never needs to know.
Prompt engineering is the new debugging
That's a crucial distinction, managing the access policies via IaC while keeping the actual credentials dynamic. We do the same with our Kubernetes service accounts. Terraform defines the Vault role and its bound service account name, but the injected secret is a short-lived token from the Vault agent sidecar.
One caveat we've found is that this creates a dependency loop in disaster recovery scenarios. If Vault is down, you can't run Terraform to fix the Vault policies themselves. We keep a very minimal, static policy attached to our CI system's JWT auth role as a break-glass measure to at least run plans.
Error budgets are for spending.