I have been conducting a detailed latency analysis of our endpoint telemetry pipeline and have encountered a persistent, performance-impacting anomaly that appears as a state discrepancy within the Trend Micro Vision One console. Specifically, multiple sensors across a heterogeneous fleet (Windows Server 2019, Windows 10 22H2) are reported as 'Disconnected' in the Vision One 'Endpoints' view, while the local `tmawf` and `tmas` services are unequivocally running and consuming expected CPU/memory resources. This state desynchronization introduces significant noise into our monitoring and incident response metrics.
The core issue is not a complete loss of data—some events still trickle into the Workbench—but the connection heartbeat appears to be failing or being discarded. This suggests a failure in the persistent, low-latency control channel, distinct from the bulk event upload channel. From a networking perspective, I have ruled out the obvious culprits:
* **Local Firewall/IDS:** We have verified the necessary egress rules for `tmawf.exe` to `agents.trendmicro.com` (TCP/443) and the regional collector FQDNs are in place. No asymmetric routing is present.
* **Proxy Configuration:** The sensors are configured for direct internet egress; no proxy is involved in these specific cases.
* **DNS Latency & Resolution:** Resolving the collector hostnames from the affected endpoints returns consistent, low-latency results (<30ms) with no packet loss. The IPs are correct and match the Vision One documentation for our region.
* **TCP Connection Health:** A continuous raw socket test to the collector IP:443 from affected hosts shows a stable connection with a consistent 0.1% packet loss rate, which is within acceptable thresholds and should not cause a 'Disconnected' state.
This leads me to hypothesize the fault lies in the application-layer handshake or credential renewal process. I have gathered the following diagnostic artifacts from a representative affected host:
**1. Service Status (Elevated PowerShell):**
```powershell
Get-Service -Name "tmawf", "tmas" | Select-Object Name, Status, StartType
```
Output confirms both services are `Running`.
**2. Recent Agent Logs (C:Program FilesTrend MicroVision One Agentlogs):**
Scouring `agent.log` and `trace.log` reveals a repeating pattern of seemingly successful collector communication interspersed with ominous but vague warnings. The key snippet:
```
[WARN] [2024-05-15 14:23:17 UTC] [HeartbeatThread] Failed to receive valid ACK from collector within timeout. Retrying...
[INFO] [2024-05-15 14:23:45 UTC] [UploadThread] Successfully uploaded 12 events to .
```
This indicates a bifurcated channel state: the event upload channel is functional (hence sporadic Workbench events), but the dedicated heartbeat/control channel is impaired.
**My primary questions for the community are:**
* Has anyone performed a packet-level analysis (e.g., Wireshark with `ssl.key_log_file`) to isolate the exact TLS handshake or heartbeat request/response failure between a 'running' but 'disconnected' sensor and the Trend Micro backend?
* Are there known, specific registry keys or configuration files (beyond the generic `agent.ini`) that govern the heartbeat interval, timeout thresholds, or collector failover logic that may be corrupted?
* Could this be a certificate (client-auth) renewal failure on the agent side? If so, what is the precise cleanup and forced renewal procedure that does not require a complete sensor reinstall?
* Is there a deterministic method to force a full state resynchronization from the agent to the Vision One console, beyond the standard service restarts (`net stop`/`net start`) which we have attempted multiple times?
The operational impact is a degradation in our real-time security posture visibility. Any detailed, technical insights into the agent-collector protocol, especially concerning state management and fault detection, would be invaluable.
Every microsecond counts.
Yeah, I've seen this exact disconnect state before when the heartbeat channel gets blocked, even with services running. It can be maddening because the event channel uses a different, more tolerant retry logic.
Since you've ruled out the network basics, check the local agent logs. The key is usually in `tmawf.log` around the `SrvReportStatus` entries. If those heartbeat attempts are timing out or getting a specific HTTP error (like a 403 from a stale token), it'll show there. I've had it happen after a regional collector had a TLS certificate change that the older agent versions didn't handle gracefully. The event upload kept retrying on a different path, but the control channel just gave up.
You could also try a forced re-registration from the agent CLI to refresh its endpoint token, but that's a bit nuclear. The logs are your best bet first. Good luck.
Automate all the things.
Interesting point about the different retry logic between channels. The forced re-registration you mentioned, is that done through the tmashell command? I'm hesitant to go that route on production systems without clearer triggers from the logs.
You're overcomplicating it. The key isn't network, it's the agent's internal state. If the heartbeat channel is dead but events flow, your agent holds a bad session token or can't parse the control response.
Check the `tmawf.log` for `SrvReportStatus` like user609 said, but look for the specific error code. I've seen this with a malformed API response after a backend update that older agent builds choke on.
Don't run a full re-registration yet. Force a heartbeat cycle from the CLI with the diagnostic command. If that fails, then you've isolated the control channel.