We're running a ServiceNow GRC instance for vulnerability management, and our scheduled scans (using the Qualys integration) have been failing intermittently. The pattern is consistent: after about 5-7 days, the MID Server stops processing scan jobs. The logs show connection timeouts or jobs stuck in "Queued" state. A restart of the MID Server service immediately resolves the issue for another week.
Our setup:
- MID Server version: 12.x (auto-updated)
- Running on a dedicated RHEL 8 VM (4 vCPU, 16GB RAM)
- Persistent connection to our ServiceNow instance
I've reviewed the standard culprits:
* No obvious memory leaks in `wrapper.log` (stays under 2GB heap).
* Network connectivity to both ServiceNow and our internal qualysgateway.com endpoint is stable.
* The `mid.log` shows no fatal errors, just a gradual increase in "Processing time for topic '...' took ... ms" warnings before it stops consuming.
Has anyone else encountered this kind of degradation pattern? It feels like a resource leak—perhaps in a connection pool or thread pool—that isn't being cleaned up. I'm considering writing a cron job to bounce the service weekly as a stopgap, but I'd prefer to identify the root cause.
Potential avenues I'm exploring:
* Enabling more verbose logging on the MID Server's JVM to monitor thread count and GC behavior over the week.
* Comparing the configuration (`mid.instance.properties`) with a known-stable older version.
Any diagnostic steps or specific log entries I should be focusing on? I'm particularly interested if others have traced this to a specific Java component or a configuration tweak that resolved it.
-- latency
sub-100ms or bust