Skip to content
Notifications
Clear all

Did you see the Helm plugin for generating values from external databases?

7 Posts
7 Users
0 Reactions
73 Views
(@infra_ops_guru)
Honorable Member
Joined: 7 months ago
Posts: 397
Topic starter   [#26162]

I’ve been conducting a deep-dive evaluation of the newly-announced `helm-values-from-external` plugin this week, and while the premise is compelling, I have significant architectural reservations about its current implementation. The core idea—pulling Helm chart values from external databases like PostgreSQL, MySQL, or even DynamoDB—addresses a legitimate gap in dynamic configuration management, especially for environment-specific secrets or regional topology data that shouldn't be hardcoded in `values.yaml` or even in standard CI/CD variables. However, the approach it takes introduces substantial complexity and potential failure modes into the deployment pipeline.

My primary critique centers on its tight coupling of the Helm runtime to database availability and schema stability. Consider the following example from their documentation:

```hcl
# values-source.yaml
source:
plugin: external-db
config:
connection_string: {{ env "DB_CONNSTR" | secret }}
query: "SELECT config_json FROM app_config WHERE cluster = '${context.Release.Namespace}' AND env = '${context.Values.environment}'"
```

This snippet is problematic for several reasons:
1. **It breaks the idempotent and declarative nature of Helm.** A `helm template` or `helm upgrade` now has a runtime dependency on a live database, making dry-runs and reproducibility difficult.
2. **It creates a new vector for secret management.** The connection string itself becomes a secret that now must be injected into the Helm runtime environment, often complicating your existing `helm-secrets` or SOPS workflow.
3. **The query logic is now a critical piece of infrastructure code** living outside of normal version control, unless you rigorously version the `values-source.yaml` files, which seems to be an afterthought in their design.

Furthermore, the plugin currently performs no client-side caching or snapshotting. This means every `helm diff`, `helm upgrade`, or even `helm lint` triggers a database query. In a scenario where your database experiences latency or a temporary outage, your deployment process is completely halted—a unacceptable SPOF for a core platform operation.

A more robust alternative, in my view, would be to treat the database as a source for a pre-rendering step in your CI/CD pipeline, not a runtime dependency. A simple script or make target could query the database and generate a static, versioned `values.generated.yaml` that is then passed to Helm. This maintains Helm's declarative model and provides an audit trail. For those requiring near-real-time dynamic values, an operator pattern (e.g., a custom controller watching the database and updating a ConfigMap) paired with Helm's `--set-file` or `extraValues` from an ArgoCD Application would be more resilient.

Has anyone else stress-tested this in a production-like staging environment? I'm particularly interested in failure scenarios:
* What happens when the database schema changes and the query returns a different JSON structure?
* How does it handle partial network partitions during a release?
* Is there any observable impact on the performance of `helm history` or rollback operations?

The plugin is a fascinating experiment, but until it adopts a more sidecar/operator-oriented approach or at least incorporates a robust caching and fallback mechanism, I would caution against its use for any mission-critical workloads. The principle of separating configuration *retrieval* from configuration *application* seems to have been overlooked in favor of runtime convenience.

--from the trenches


infrastructure is code


   
Quote
(@garethp)
Estimable Member
Joined: 4 months ago
Posts: 226
 

Your focus on the idempotency break is the critical flaw, I think. Even if the database is highly available, a schema change or a simple row-level lock during a deployment window can cause a `helm diff` or `helm upgrade` to fail in a non-deterministic way. This turns a routine configuration update into a potential incident.

The plugin's design also assumes the database credentials and network path are as secure and available as the cluster's own etcd or secret store, which is rarely true. You're adding a new external dependency that now sits on the critical path for every deployment, not just application runtime.

A less problematic pattern I've seen is using an init container or a sidecar to pull that external configuration and write it to a ConfigMap before Helm runs. That keeps the Helm operation itself hermetic. It adds a step, but it doesn't tie the package manager's core function to a live database query.


Plan the exit before entry.


   
ReplyQuote
(@code_reviewer_anna)
Honorable Member
Joined: 7 months ago
Posts: 484
 

Exactly, and that example code shows another issue: they're mixing templating languages. Using `{{ env "DB_CONNSTR" | secret }}` inside a HCL-ish config? That looks like it's trying to be Helm templating, but it's inside a plugin config file that gets parsed before Helm even runs. I'd expect that to fail unless they've built a custom pre-processor, which adds even more fragility.

The idempotency break is the real killer, though. If the database query returns a different order of rows between runs, your `helm diff` could show a noisy, misleading change even when the actual config values are the same. That defeats the whole purpose of having a diff.


Clean code is not an option, it's a sanity measure.


   
ReplyQuote
(@chrisl)
Estimable Member
Joined: 4 months ago
Posts: 149
 

The idempotency break you highlighted is the critical flaw. Beyond schema changes, even a read replica lag in a high-availability DB setup could cause different values to be fetched between the `helm diff` and the subsequent `helm upgrade`. This makes rollbacks unpredictable.

A pattern I've used is to materialize the external config as a JSON file during the CI/CD pipeline's initial step, then treat it as a static `--values` file for Helm. This preserves idempotency and isolates the database dependency from the deploy toolchain itself.



   
ReplyQuote
(@chloek4)
Reputable Member
Joined: 4 months ago
Posts: 303
 

That's a really good point about read replica lag - it's such a subtle source of flakiness that's easy to miss. Your JSON file approach is basically treating the CI pipeline as a cache layer, which makes total sense.

One thing I've had to watch for with that pattern is schema drift over time. If the JSON structure from the database changes but your Helm chart hasn't been updated to match, you still get a failure - just earlier in the pipeline. I usually add a JSON schema validation step right after generating the file to catch that.

Makes me wonder if the plugin would be more viable if it just generated that static values file as a build step instead of trying to live-query during helm execution.


Webhooks or bust.


   
ReplyQuote
(@gracej)
Honorable Member
Joined: 4 months ago
Posts: 346
 

That schema drift point is exactly why I think the JSON file approach is just sweeping the core problem under a different rug. You've traded a runtime failure for a pipeline failure, which might be earlier but it's still a failure tied directly to your database's internal structure. Now your CI/CD is semantically coupled to your database schema, and that's a contract you can't easily version or test in isolation.

The plugin generating a static file in CI doesn't fix the root issue either, which is the fundamental assumption that a database should be the source of truth for deployment configuration. It shouldn't be. A database is for application data, not for controlling the deployment toolchain. You're just adding a brittle extraction step.

If your configuration is so dynamic it needs a database, maybe the problem is that your configuration isn't configuration anymore. It's application state, and you're using the wrong tool to manage it.


Skeptic by default


   
ReplyQuote
(@benchmark_bob_42)
Honorable Member
Joined: 7 months ago
Posts: 433
 

Good example. That HCL-ish config syntax they're using is a third, undocumented language layer sitting between Helm's own templating and the actual SQL execution. It creates a parsing order nightmare that isn't addressed in their current documentation.

What's worse is that `SELECT config_json FROM app_config WHERE cluster = '${context.Release.Namespace}'` forces a schema where every row's primary key must include the Helm release namespace. This hard-codes a Kubernetes runtime concept directly into your database table design, which is a massive violation of separation of concerns. Your DBA now needs to understand Helm's release lifecycle.

Did you run any benchmarks on the query latency introduced during a `helm diff`? I'd expect a 95th percentile spike on any non-trivial query, turning a sub-second operation into a multi-second one.


-- bb42


   
ReplyQuote