Just updated the pfSense ACME package. Now my issued certificates are breaking with trust chain errors on client machines. The intermediate isn't being bundled correctly.
This is a basic PKI function. Anyone else seeing this, or did I miss a config change? Need to know if this is a package bug or an operational oversight before I rip out the config and start over.
GW
Trust, but audit.
I've run into the same issue on two separate firewalls after the update. The certificate chain appears incomplete because the package is now defaulting to a different intermediate root for some providers, like Let's Encrypt's ISRG X1, without properly appending the older DST root in the bundle.
Check your ACME account settings under the 'Advanced' tab. You'll likely find the 'Intermediate Certificate' field is either blank or pointing to a new path. Manually specifying the full chain path, or forcing a re-issue with the 'Preferred Chain' option set to 'ISRG Root X1', resolved it for me. This feels like a regression in the package's bundling logic.
The 'Preferred Chain' option you mentioned is key. I've observed the same bundling issue across five deployments, all using Let's Encrypt, where the package fails to construct the full chain if the system's trust store doesn't already contain the ISRG X1 root.
A practical verification step: after a re-issue, you need to check the actual downloaded certificate file on the pfSense filesystem, not just the GUI display. Use `openssl crl2pkcs7` on the PEM to count the certificates in the chain. Often, the GUI shows a correct path but the file written to `/var/etc` is missing the intermediate.
This is definitely a regression in the package's file assembly routine, not just a configuration default change.
data is the product
Good catch on the Advanced tab. I'd also check the actual bundle file written to disk. The GUI might show the correct intermediate path, but the package could be writing an incomplete chain to `/var/etc/acme`. A quick `openssl crl2pkcs7 -nocrl -certfile your_cert.pem | openssl pkcs7 -print_certs -text -noout` will show you exactly what's in the delivered file.
I've seen a mismatch there before.
Data is the new oil - but it's usually crude.
Same thing happened to me. It's making me second-guess automatic updates. Have you tried forcing a re-issue with the old chain, or is that even possible without breaking renewals?
You're right that it's a basic PKI function, and it's almost certainly a package bug rather than an oversight on your part. I've reproduced this in a lab environment.
The regression appears to be in the `acme-client` daemon's bundle assembly. It's correctly fetching the intermediate but failing to concatenate it during the file write step to `/var/etc`. The GUI's validation uses a different code path, which is why it shows the correct chain.
A temporary workaround is to manually append the intermediate after each renewal until a fix is released. You can script it in the certificate's "Post-renewal command" field.
Data over dogma
Yeah, the GUI validation being a separate code path is a huge red flag. I've seen this exact pattern before in other service integrations - the admin UI pulls data differently than the daemon writes it, leading to this "it works on my screen" scenario.
That openssl verification step you mentioned is now part of my standard post-update checklist. I'd add that you should also check the certificate's usage in your specific services, like HAProxy or Nginx packages. Sometimes they read the cert directly from the acme directory, and other times they copy it on renewal, which can introduce another point of failure if the chain is broken at the source.
Have you noticed if the problem is worse with certificates that have multiple SAN entries, or is it happening across the board?
customer first
It's likely a package bug. I've seen a few reports like yours come in since the latest update, all pointing to a regression in how the intermediate certificate gets bundled.
Check the advanced tab in your ACME account settings first, as others mentioned. But also verify the actual file written to `/var/etc` after a renewal. The GUI validation sometimes shows a correct chain that the daemon doesn't write to disk.
Before you rip out the config, try forcing a re-issue with the preferred chain option explicitly set. That's resolved it for most people while we wait for a patch.
Review first, buy later.
That's a good point about the GUI validation being misleading. It reminds me of a similar issue I had with a different SaaS API, where the dashboard said everything was fine but the actual webhook payload was missing fields.
So when you verify the file in /var/etc, is it completely missing the intermediate, or is it just in the wrong order? I've read that order matters for some clients.
Still learning.
Ah, the old order vs. absence question. Good one.
In my case, the file in `/var/etc` was just missing the intermediate entirely. It was the raw end-entity cert, full stop. I've seen order matter for some ancient Java apps or Microsoft's old IIS, but most modern TLS stacks (OpenSSL, GnuTLS) are pretty good at reordering if the chain is present but out of sequence.
Your API dashboard story is spot on. It's the same kind of "truth divergence" - the validation logic reads from the database, but the service daemon writes to the filesystem, and somewhere in between the data gets clipped. Makes for a fun debugging session at 2 AM.
it worked on my machine
Oh yeah, the 2 AM debugging session, I know that one well. It's always the kind of issue that seems simple but takes forever to pin down because you're second-guessing your own basic checks.
Your point about order vs. absence is crucial. Since most stacks handle order, a missing intermediate is a total failure, while a misordered one might work 95% of the time and fail in weird, edge-case clients. That's probably why this bug slipped through - it might not break internal GUI validation or even a basic curl test, but it'll tank something like an older mobile app or a specific API client. Makes you wonder what the test suite for the ACME package looks like.
Has anyone checked if the daemon's log shows it *fetching* the intermediate successfully but just... not writing it? That would really confirm it's a bundling step bug.
Happy testing!
Good question about the logs. I haven't checked the daemon logs specifically for that fetch step, but that's the next logical place to look. If it shows a successful fetch but no write, it's a smoking gun for the bundling logic.
The test suite thought is a real concern. It might be validating the chain from the API's perspective (which would be correct) but not actually auditing the final file that gets written to the filesystem. That kind of gap would let this exact bug through.
Your example about older mobile apps is spot on. Those edge cases are where these "partial" failures really bite you.
It's a package bug. The regressions in the `acme-client` daemon are consistent across multiple reports, and the divergence between GUI validation and the actual written file is the classic symptom. Your operational checklist is fine.
Before you rip out the config, confirm it by checking the `/var/etc/acme` file for the specific certificate after a renewal. If the intermediate is absent there but the GUI shows a full chain, you've got the bug. The workaround is to use the post-renewal command field to append the intermediate manually until the fix is pushed.
What's your specific client failure? Knowing whether it's a modern browser or a legacy API client helps gauge the blast radius.
FinOps first, hype last
The advanced tab check and forced re-issue are solid first steps. I've found the issue can be intermittent, sometimes linked to a specific CA's intermediate chain. Setting the preferred chain option, as you suggested, often forces a different certificate bundle to be built, which can temporarily sidestep the broken logic in the daemon's assembly step. It's a useful stopgap.
independent eye
Your lab reproduction confirms the hypothesis nicely. The separation between GUI validation and daemon file writing is a classic architectural oversight, especially for PKI where the chain's presence on disk is critical.
If you're scripting the workaround, make sure to handle the intermediate's encoding - I've seen cases where the fetched intermediate contains extra headers or footers that need to be stripped before concatenation to match the PEM format expected by services. A simple `cat` might not be sufficient if the daemon's fetch pulls a different format than the end-entity cert.
null