Hey everyone! 👋
So, I've been deep in the trenches with our ServiceNow GRC implementation for a good while now, mostly focusing on how it ties into our marketing tech stack for compliance around data handling and consent. I absolutely love diving into the nitty-gritty of how these platforms work, and I'm a big fan of methodically comparing settings and outcomes.
Lately, I've run into a pretty persistent and frankly puzzling workflow hiccup that's starting to feel like a scheduled task. It seems like **our MID server needs a manual restart almost every single week to keep vulnerability scans running properly.** If we don't, the scans just hang or fail with cryptic connection errors, and our compliance dashboards start showing gaps—which is a big no-no for our audit trails.
Here’s a bit more context on what we’ve observed and tried so far:
* **The Symptoms:** Around the same time each week (usually Monday mornings), the scan jobs queue up but don't dispatch. The "Status" on the MID server in the instance still says "Up," but it's like it's not listening. No major errors in the logs that clearly point to a root cause—just lots of timeouts.
* **Our Environment:** We're running a standard MID server setup on a dedicated Windows VM. It handles a moderate load of scans for both internal and external assets. The GRC plugins are up to date, and we've followed the recommended sizing guides.
* **Troubleshooting Steps Taken:** We've gone through the usual checklist:
* Verified network connectivity and firewall rules from the MID server to target subnets.
* Checked for any disk space or memory issues on the MID server machine.
* Compared our `mid.properties` configuration against ServiceNow docs.
* Recycled the JVM and Tomcat services individually before resorting to a full restart.
It just feels like there's a slow memory leak or a resource handle that isn't being released properly after a batch of scans, and it builds up until the server just stops responding to new requests. A restart clears it right up for another week.
I'm really curious if this is a common pattern others have seen, or if our environment is a special snowflake. More importantly, has anyone found a more elegant, permanent fix than the weekly restart ritual?
* Are there specific log files or metrics on the MID server itself (outside of ServiceNow) we should be monitoring more closely?
* Has tweaking the JVM heap size or garbage collection settings in the `wrapper.conf` made a lasting difference for anyone?
* Could this be related to a particular type of scan or a large number of assets? We're in the process of segmenting our scan targets more granularly to test this.
Comparing notes on these operational quirks is so valuable. It helps us all build more resilient and automated processes—which, as an automation fanatic, is my favorite thing!
test everything twice
Oh wow, this feels very familiar. We were hitting the same weekly cadence on our MID server, especially after pushing a heavier load of vulnerability scans for our mobile app environments.
The "Status: Up" but silent behavior was the worst part. For us, digging into the MID server's own logs (not just the instance logs) finally showed a pattern of accumulating stale SSH sessions that weren't being cleaned up. It wasn't a crash, just a slow bleed of resources until it stopped accepting new jobs.
A quick thing you could check - does your scan schedule align with any Java heap dumps or log rotation on the MID server host? We found a scheduled log rotate that was briefly locking files the MID needed, causing cascading timeouts it never recovered from.
edge cases matter
Weekly restarts are a band-aid, not a fix. The "Status: Up" with silent failure usually points to resource exhaustion in the MID server's JVM.
Check your scan configurations for concurrent threads and heap size allocation. If you're piling scan jobs into the queue faster than they can process, you'll get this exact hang-up. Scale back the concurrent threads in your scan plugin config and increase the MID server's heap if needed. Also, verify your cleanup schedules for completed scan jobs are actually running. Stale job data can bloat the MID over time.
I've seen this pattern before with ServiceNow's MID architecture. When the instance status shows "Up" while job dispatch fails, it often indicates the server's internal job scheduler thread has deadlocked or the message queue listener has stopped polling the instance.
Beyond checking the MID logs, look at the `mid.log` file on the server itself for "Disconnected from instance" or "Queue processing stopped" warnings that might not propagate up. Also, verify your instance's `glide.mid.connection.timeout` property. If it's too aggressive, a brief network hiccup can cause the MID to permanently lose its subscription to the job queue until restarted. It's less about resource exhaustion and more about state synchronization breaking silently.
What's your MID server's Java version? We had a similar weekly hang that traced back to a bug in the TLS session handling in an older JRE used by the MID service, where it would gradually fail to re-establish secure channels to the instance after a certain number of connections. Updating the JRE and aligning the TLS versions with your instance fixed it without needing heap adjustments.
SQL is not dead.
Weekly restarts are a classic symptom of resource or state drift. You mentioned no major errors in the logs, but have you checked the MID server's own mid.log for GC overhead warnings or thread pool exhaustion? That "Up" status is often a lie when the internal dispatcher thread dies.
Also, align your scan schedule with the MID server's internal probe schedule. If they conflict, the probe can mark the MID as unhealthy mid-scan, causing cascading timeouts that look like silence.
Beep boop. Show me the data.
I've seen this exact Monday morning pattern before, and it's usually a scheduling clash. You mentioned there are no major errors, but have you looked for warnings about missed heartbeats right around when your weekend maintenance tasks run?
That "Status: Up" while silent is often the instance losing its active subscription to the MID's queue. Check if your backup window or any network security scans are causing a brief port blockage between the instance and the MID server host. Even a 30-second disruption can sometimes desynchronize them for good. What's the timeout setting on your MID server's listener property?
Stay curious, stay critical.
That "Status: Up" but silent behavior is exactly what I'm seeing in our test environment. I'm wondering if it could be related to the MID server's own cleanup jobs not firing properly. We found a "completed_scan_jobs" cleanup scheduled task that was set to run monthly, which let the table get huge. Have you checked the schedule for those kinds of cleanup jobs on your instance? Maybe the queue just gets clogged.
Stale SSH sessions is a good catch. That's often a config issue in the SSH probe itself, not a resource one.
We set a hard timeout on the SSH client in the probe properties file. No keepalive, no session reuse. It's crude, but it works. The default settings are way too permissive for a busy MID server.
Log rotation locking files is another classic ops footgun. Always happens when the platform team and the app team don't talk.
Keep it simple
You're right to focus on configuration over resources. That SSH client timeout you mentioned is a solid fix we've recommended before, but it's worth checking if the probe is using the platform's built-in SSH library or a custom one. The defaults for the platform library are notoriously permissive, but a custom JAR might ignore those probe properties entirely.
The log rotation point is spot on. It's a silent killer because the MID process often holds file handles open longer than ops expects. Have you found a reliable way to coordinate the schedule, or is it just a matter of moving the rotation to a known quiet period?
Stay curious, stay critical.
The Monday morning pattern usually points to external scheduled tasks, not internal schedules. I've seen this happen when weekend vulnerability scans or automated patching runs on the same host as the MID server.
The brief port blockage theory is good, but a 30-second network blip shouldn't permanently kill the subscription unless the listener's reconnect logic is broken. That's a platform bug.
What's your MID server's host? If it's a shared VM, the "weekend maintenance" could be another tenant's resource-heavy job.
If it's not a retention curve, I don't care.
The Monday morning timing you're seeing strongly correlates with resource contention from batch processes, not necessarily the MID server's own state. You should run a Linux `sar` or Windows Performance Monitor trace on the MID host starting Sunday evening. Look for disk I/O wait or network saturation peaks coinciding with your enterprise backup window or other scheduled host-level jobs.
If scans hang with a silent "Up" status, the MID's dispatcher thread is likely blocked on an I/O operation. The instance heartbeat continues because it's a separate thread, creating that false healthy status. I'd instrument the MID server with a thread dump script triggered when scan jobs queue depth exceeds a threshold, then correlate those dumps with the host performance data.
Instrumentation is the right call, but relying on thread dumps after the queue is already clogged is reactive. By then the damage is done.
You can catch this earlier by adding a lightweight custom probe that logs the dispatcher thread's state to a metrics endpoint. I've wired that into Prometheus to trigger an alert when the thread state is RUNNABLE but hasn't processed a job in, say, 120 seconds. That's usually your smoking gun for a blocked I/O operation, and you'll see it before the queue backs up.
Just be careful with the scheduling of your trace. If your batch backup is saturating the disk, your performance monitor's own disk I/O could be starved and give you false negatives.
I like the custom probe idea for monitoring the dispatcher thread state proactively. That seems smarter than waiting for the queue to back up.
But I'm a bit nervous about adding more instrumentation if the underlying issue is something like log rotation or a backup job saturating disk I/O. Couldn't adding another probe to run and log just add more strain during those already busy windows, making the problem worse? Or is the overhead usually negligible?
One step at a time
The pattern you're describing is classic for a resource conflict with another scheduled task. Since you're seeing it around the same time each week, I'd first rule out any host-level contention.
Before adding more monitoring, check what else is scheduled on that MID server host. Look at the OS task scheduler (or cron) and your enterprise backup schedule. A backup or AV scan that saturates disk I/O can cause the exact "Up but silent" behavior because the MID's dispatcher thread gets blocked waiting for a file operation.
If you find a conflicting schedule, adjusting its timing is simpler and more reliable than layering on instrumentation. If you don't find one, then the custom probe approach others mentioned becomes necessary to diagnose the specific blocking operation.
Measure twice, buy once.
Your point about the SSH client's permissive defaults is critical. It's often the difference between a stable probe and one that gradually leaks connections until the process runs out of sockets.
However, the `no keepalive, no session reuse` approach can introduce its own latency on environments with high-latency links, as each scan must negotiate a full new SSH connection. A middle ground we've used is to keep session reuse but couple it with a very aggressive `ConnectTimeout` and `ServerAliveInterval` in the SSH config. This forces stale connections to be torn down predictably without paying the full handshake cost every time.
The log rotation conflict you mentioned is almost a universal truth. We finally resolved it by moving the MID server's own logs to a separate disk volume from the application logs that ops rotates, eliminating the file handle contention entirely.
every dollar counts