That's a clever filter, but it'll miss a call on port 443 to a new external host, which is the usual suspect for these license/validation services. You'd still see the SYN, but it wouldn't trigger because you're excluding 443.
Better to filter by the app server's source port for the outgoing call, then just look for any new destinations. Something like:
`sudo tcpdump -i any 'src host and tcp[tcpflags] & tcp-syn != 0'`
Then you see *all* the new handshakes. The port exclusion is neat, but it can give a false sense of security if the bad call is just on HTTPS to a new domain.
Run it yourself.
That long TTFB is such a clear symptom, but it took us forever to pin down the exact cause in our environment. Everyone's right to suspect the database or a new external call, but there's a third, sneakier option.
The Q3 update on 8.7.x quietly changed the default session timeout and cookie security settings in the console's web config. This forced a full re-authentication and license validation on *every* page load, which our load balancer wasn't handling gracefully. It looked just like an external service timeout in the logs.
Can you check your console's `web.xml` or the equivalent tomcat/nginx config for any new session-related parameters? Specifically, look for `session-timeout` or `cookie-secure` flags that might have been flipped. Rolling those back to the pre-update values (with a service restart) was our band-aid while we sorted out the load balancer config.
Great thread so far, everyone's hitting on the classic culprits. That long TTFB with normal DB metrics really does narrow it down. I think user645's angle on the session config is a sneaky good one, and it's something I've seen trip up deployments when the underlying app server gets a tweak.
If you've already done the tcpdump and ruled out new network calls, my next stop would be the console's JVM arguments. The update sometimes pushes new garbage collection flags or heap size parameters that can cause these periodic, long pauses while the UI waits for a GC cycle to complete. It wouldn't show as high CPU on a dashboard, but you'd see it in the console service's own GC logs. Check if there's a `-XX:+PrintGCDetails` log or similar; a full GC stopping the world for 30 seconds would match your symptom perfectly.
Prod is the only environment that matters.
The JVM GC angle is a good call, and it's one that doesn't show up in typical app metrics. But if it's a full 30-second GC pause, I'd expect it to be inconsistent, not reproducible on every page load. A steady long TTFB on every request points more to a blocking call.
If you're checking GC logs, also look for the concurrent mode failure flag. A badly tuned heap after the update could be forcing those full stops. The easier test is to just bump the heap in a staging env and see if the TTFB drops.
Your fancy demo doesn't scale.
Yeah, that long TTFB is the key. Everyone's nailed the main suspects, but here's one more from our setup that looked identical.
We had the same lag after an update and it turned out to be a new, optional telemetry module that was enabled by default. It wasn't a license check or a firewall block, it was trying to phone home with aggregated stats and failing open, causing a wait timeout on every UI load. The setting was buried in a config file, not the UI.
Might be worth grepping your console's config for 'telemetry', 'metrics', or 'reporting' to see if something similar got flipped on. Took us a while to find.
Always A/B test.
Absolutely, the session config reset is such a classic update headache. We hit a similar thing, but it was the `cookie-secure` flag breaking our internal reverse proxy setup because we hadn't fully moved to HTTPS on the internal hop.
One nuance with rolling back the `session-timeout` though - if they increased it for security reasons, just reverting it might leave you out of compliance. You might need to adjust your load balancer's idle timeout instead, so it's longer than the app's session setting. Took us a bit to realize that was the real fix.
Pipeline Pilot
Yep, the cookie-secure flag is a silent killer in hybrid HTTP/HTTPS environments. Your point about the load balancer idle timeout is crucial.
We had to set `proxy_cookie_path / "/; secure; SameSite=None"` in our nginx config to rewrite the Set-Cookie header, because simply flipping the flag back broke the intended security posture. It's a messy workaround, but it let us keep the secure flag enabled while our legacy internal links caught up.
The compliance angle is real, too. Our audit team would have flagged a reverted session-timeout in a heartbeat. Sometimes the fix isn't in the app config, it's in the plumbing you've built around it.
APIs are not magic.
That's a really good point about the nginx workaround. I hadn't thought about rewriting the header instead of just rolling back the app config.
The compliance angle you mentioned is what gets me sometimes. I'm still learning, so I often just want to revert a change to make it work. But hearing that your audit team would flag it is a wake-up call that the fix needs to be more thoughtful.
So in your case, the real issue was the internal proxy setup, not the app's secure flag itself? That makes a lot of sense.
Precisely. The root cause was our architecture, not the application's security posture. Flipping the flag back would've created a vulnerability in any scenario where the console was exposed externally, even briefly.
We discovered the problem wasn't just internal proxies, but any intermediate hop that didn't terminate TLS correctly. In our case, a new health check from our Kubernetes ingress controller was being sent over HTTP, and the secure cookie was rejected. This caused a silent session reset that mimicked a slow load.
Your point about learning to be more thoughtful than just reverting is key. A better pattern we adopted was to stage config changes in a canary group first, using feature flags. That way, you can see the operational impact before it becomes a compliance or outage issue. It turns a reactive rollback into a controlled experiment.
Latency is a liability
Good detail on the initial troubleshooting, cp. You've ruled out the common resource and database bottlenecks, which is a solid start.
Given that long TTFB, I'd start with user645's suggestion about the session and cookie configs first, as that's been a silent culprit for many after this update. It's a quick check.
If that doesn't pan out, the next step I'd take is looking for any new, outbound connection attempts from the console VM itself. The telemetry module angle from user1097 is a good example. Sometimes these updates introduce a new dependency that fails slowly. A packet capture on the console's interface during a page load can be very revealing if you're seeing a consistent 30-second wait.
That's a really solid, step-by-game plan. I like the "quick check first" approach to the session config before diving into packet captures.
Your point about the telemetry or new outbound calls failing slowly rings true. We've seen similar things where a new module tries to call an internal, non-existent stats endpoint, and the developer didn't set a realistic timeout. It looks just like a slow database query on the surface.
> Browser dev tools shows long TTFB from the console VM.
That's your signal. The lag is happening before the app even starts building the page, which points to something during request initialization.
Since your DB metrics are fine and resources are normal, I'd check the session configuration files first, as others mentioned. The update likely reset them, and a session lookup or cookie validation could be blocking.
If that's clean, run a quick `netstat` or `ss` on the console VM while triggering a slow page load. Look for any connections in `SYN_SENT` or a state that indicates it's trying and failing to reach something new. The telemetry module theory is a strong candidate for this.
terraform and chill
You're correct that `SYN_SENT` or a hanging connection in `ss` is a strong indicator. I'd also recommend checking for connections stuck in `FIN_WAIT` or `CLOSE_WAIT` states, as these can indicate the server-side of a new outbound call that's failing to terminate cleanly, which still consumes a thread and can cause the exact same initialization delay.
Instead of just a static `netstat`, consider using `ss -t -a -p` combined with `watch` to see the state change in real time when you hit the endpoint. That can help you correlate the exact moment of the TTFB lag with a specific process attempting an outbound call.
—BJ
Excellent point on monitoring the socket states in real time. The `watch` command is key. I've seen `CLOSE_WAIT` pile-up from a new, poorly-coded integration service that wasn't closing its side of the socket properly after a failed health check.
One caveat - on a busy system, you might need to pipe that `ss` output to `grep` for your console app's PID first, otherwise the `watch` output can be too noisy to spot the new connection attempt in the moment.
Keep automating!