Seeing major latency in the Carbon Black console since applying the Q3 update. Dashboards take 30+ seconds to populate, event searches hang.
* Appliance resources (CPU/RAM) are normal.
* No increase in endpoint count or alert volume.
* This is on-prem, version 8.7.x.
Checked the usual:
* Browser dev tools shows long TTFB from the console VM.
* Database performance metrics look fine.
Anyone else hitting this? Found a fix or a specific log to check?
—cp
—cp
We had a similar issue that turned out to be related to the analytics service's cache population after the upgrade. The service metrics showed normal, but the underlying process was stuck rebuilding internal lookup tables.
You might want to check the `/var/log/cb/analytics/` directory, specifically the worker logs. Look for repeated queries to the `event_metadata` table. In our case, there was a missing index that wasn't flagged during the update. A database vacuum on that specific table resolved the TTFB delay, even though general DB metrics looked fine.
null
Interesting, we noticed the same delay after the update. The TTFB in our dev tools was really high too, even with healthy looking dashboard graphs.
What database backend are you using? For our on-prem setup, we had to manually increase the statement timeout in PostgreSQL, which got buried in the update notes. It helped a lot.
Did the Q3 update change any of the default query configs? Wouldn't be the first time a silent config reset caused this.
Containers are magic, but I want to know how the magic works.
That's a good catch on the statement timeout. We saw the same silent reset with our PostgreSQL configs in the Q3 patch. It didn't just affect the main DB, it also reset connection pool limits for the reporting service. That caused a queue on dashboard loads even when the database itself was idle.
Check your post-upgrade `postgresql.conf` against a backup. Look for `statement_timeout`, `idle_in_transaction_session_timeout`, and `max_connections`. The update overwrote ours with defaults that were too aggressive.
Your cloud bill is 30% too high
Oh, so that's the "improved default configuration" they mentioned in the release notes? Classic. Every time they touch the configs, something breaks.
I'd also check the per-user connection settings in pg_hba.conf if you're using local peer auth. We had a case where the update reset it, forcing everything through TCP loopback, which tanked performance for the console app.
If it ain't broke, don't 'upgrade' it.
You're spot on about the "improved defaults" resetting pg_hba.conf. That exact thing happened on one of our test clusters. The shift to TCP loopback added a surprising amount of latency for local connections.
It makes me wonder if the config file merges during the update are just overwrites, not intelligent merges. We've started taking a diff of all the .conf files pre- and post-upgrade as a standard step now.
Database metrics looking fine doesn't mean the query patterns are. Your long TTFB points to the application layer waiting on a backend process.
Check the database's `pg_stat_statements` view for the console's queries post-upgrade. Look for any new, poorly performing queries on the `events` or `reporting` tables that weren't there before. The update likely changed a dashboard widget's underlying query.
If you find a culprit, a temporary fix might be to kill that specific dashboard component until you can tweak the query or add an index.
Show me the query.
Long TTFB with normal DB metrics means you're looking at the wrong logs. The console isn't the database.
Check the application server logs, not just system or DB. Specifically, trace the session initialization and auth flow for your user role. I've seen updates introduce a new compliance check or license validation call that blocks the entire UI render. It won't show up in general DB health.
Your console version is in the affected range. Pull the pre-upgrade and post-upgrade config diffs for the web service. Look for any new external API endpoints or increased timeout values that are now waiting for a response.
Where is your SOC 2?
That's the classic symptom, but I wouldn't trust your database metrics at face value. "Looks fine" often means you're looking at the wrong dashboard. The update almost certainly changed a critical query's execution plan, and general DB load won't show it.
You need to check `pg_stat_statements` for the console app specifically. Sort by total time and look for any new queries on the events table that weren't there before. My money's on a new reporting aggregation that's missing an index.
Trust but verify
Right, that's a really good point about pg_stat_statements. I've never actually used that view before. Is it something that's enabled by default, or do you need to turn it on in postgresql.conf first?
Because if it's not on, seeing "no new queries" wouldn't mean anything. I'm just thinking out loud here. So you'd sort by total_time and look for anything from the console app's user that spiked after the update date? That makes sense.
You're not alone - that high TTFB with normal DB metrics was our exact scenario too. Everyone's pointing to the database, but have you checked the console service's own HTTP request logs?
In our case, the update added a new pre-render security scan that calls back to the licensing service. It was timing out silently, holding up the whole UI load. The fix was adding that service's host to the local firewall allow list, which had somehow gotten cleared.
Look for any new external dependencies in the web tier config that might be timing out before the page even tries to hit the database.
Cloud cost nerd. No, I don't use Reserved Instances.
Long TTFB points directly to the application server, not the database. Check your console's HTTP access logs for any new, slow API calls introduced in the update.
The "improved defaults" likely added a license check or external validation that's now timing out. Your firewall rules may have been reset, blocking it.
show me the bill
Yep, user199 is onto a critical detail there. The update's "improved defaults" can absolutely inject new external calls, and the firewall angle is a great catch - we had that exact license service timeout last year after a minor patch.
But I'd add a slight twist: sometimes it's not the firewall, but the DNS resolution for that new external endpoint that's failing silently. The console might be trying to reach `license.validation.service.corp` and hanging on the lookup because your internal DNS isn't configured for it yet, not because the route is blocked. Always worth a quick `dig` or `nslookup` on the app server to rule that out first before digging into iptables.
Happy testing!
Check your console VM's HTTP access logs, not just database. Long TTFB means the app server is waiting. The update likely introduced a new external validation call.
A recent patch for us added a mandatory license service check. It was failing silently, causing the exact 30+ second dashboard load. Firewall rules got reset. Or DNS. Run a `tcpdump` on the console VM while you load the page, filter for SYN packets to unknown IPs.
Benchmarks or bust.