Hi everyone. I’ve been quietly following the discussions here for a while, but we’ve hit an issue I haven’t seen covered yet.
Since deploying the latest agent update (v4.7) across our Windows 10 and Server 2019 systems, we’ve seen a significant spike in sensor disconnections. Our dashboard shows intermittent drops, usually lasting 2-5 minutes, before the sensors reconnect. This wasn’t happening at this scale with the previous version.
Our environment is relatively standard, with the agents communicating through our proxy servers. We’ve double-checked the proxy rules and firewall logs, and there’s no increase in blocked traffic that correlates with the disconnection times.
I’m curious if anyone else is observing this pattern after the update. Specifically:
- Are the disconnections clustered at particular times, like during system startup or a specific scheduled task?
- Has anyone found a correlation with other security software? We run a separate EDR tool, but there were no conflicts before.
- Are there any recommended configuration tweaks for the sensor settings in the policy to improve stability with this version?
I’m looking through our Carbon Black Cloud console for clues, but any pointers on where to focus the logs would be really helpful.
Thanks for raising this so clearly. You've nailed the right first steps by checking the network logs and proxy rules, which usually is the culprit. Since that's not it, I'd suggest a couple of angles.
First, check if the disconnections are clustered right after the agent's scheduled full scan kicks off, if you have that enabled. v4.7 did tweak some resource throttling logic, and on some of our older Server 2019 boxes, we saw a similar pattern where a high CPU spike from the scan would briefly break the sensor's heartbeat.
On the security software front, it's a good call. Even if there was no conflict before, your EDR might be interpreting a new behavior in the v4.7 agent's memory access or network polling as suspicious and briefly isolating the process. A temporary test with the EDR's real-time protection suspended on one test machine could rule that in or out quickly. Let us know what you find in the console logs about the disconnection reason codes.
Stay curious, stay critical.
We saw the exact same pattern across about two hundred Windows Server 2016 and 2019 nodes. Blaming the proxy or the network stack is the obvious first step, but you're right to move past it when the logs don't support that.
In our case, the root cause was the new telemetry compression routine in v4.7. During periods of high log generation, the agent would spike memory usage and hit a hidden, poorly-documented internal timeout trying to compress a batch. This caused the sensor process to hang, triggering the disconnection. The 2-5 minute window is roughly how long it took for the watchdog process to kill and restart it.
The workaround, which they've since documented in a KB article, was to add an advanced policy setting to limit the maximum event batch size. We dropped `max_batch_size_bytes` to 512000 and the disconnects vanished. It's a band-aid, but it works until they issue a proper fix. Check your agent logs for warnings about batch processing or compression timeouts; that's your smoking gun.
Interesting, I'm seeing a similar pattern on a small scale test. I was checking for CPU spikes like user793 suggested, but our monitoring doesn't show a clear match.
You asked about configuration tweaks. Did the policy setting user1339 mentioned for the `max_batch_size_bytes` work for you? I'm about to try it, but I'm a bit nervous applying it without knowing if it could cause data loss or backlog on some of our busier servers.
No offense to the other suggestions, but they're guesses. The 2-5 minute cycle is the key symptom, and it's almost always the watchdog process killing a stalled sensor thread.
user1339 is on the right track with the batch size, but that's one of several new internal timeouts that can trigger this. The v4.7 release notes were... optimistic about its resource handling.
Before you apply a policy change, check the agent's debug logs for `WATCHDOG` entries. That'll confirm if it's the compression hang or something else, like the new metric aggregation routine. If you see batch compression errors, then yes, `max_batch_size_bytes` is your fix. We set ours to 512KB and the disconnects stopped.
It doesn't cause data loss, it just forces more frequent, smaller uploads. The backlog risk is minimal unless your network is already saturated.
Your fancy demo doesn't scale.