Skip to content
Notifications
Clear all

Anyone else having massive gaps in SecurityEvent table for some Windows servers?

39 Posts
37 Users
0 Reactions
377 Views
(@code_weaver_max)
Reputable Member
Joined: 6 months ago
Posts: 370
Topic starter   [#27006]

Hey folks, has anyone else been pulling their hair out over random, massive gaps in the SecurityEvent table for specific Windows servers in Sentinel? 😅

I've got a stable environment with about 50 servers all configured identically (I thought!) via the same Log Analytics agent (AMA) and SecurityEvent policy. For most, the logs flow in like a steady stream. But for three particular servers, the `SecurityEvent` table just... stops for hours at a time. No errors in the agent health table, no obvious network blips. The System and Application logs from those same servers come through without a hitch!

I've been down the rabbit hole checking the obvious:
* Agent health (`Heartbeat` table) shows consistent pulses from the problem children.
* Local Windows Event Log on the servers confirms the events *are* being generated (I checked!).
* The workspace diagnostic settings look correct, and the `SecurityEvent` collection is enabled at the workspace level.

The only pattern I've spotted is that the gaps seem to coincide with periods of higher event volume (like during automated patching). Could it be a throttling issue on the Microsoft Monitoring Agent side? Or maybe a conflict with the local Windows Event Log size/rollover settings?

Here's the basic KQL I've been using to visualize the gaps for a specific problematic computer:

```kql
SecurityEvent
| where Computer == "ProblemServer01"
| where TimeGenerated > ago(7d)
| summarize EventCount = count() by bin(TimeGenerated, 15m)
| render timechart
```

It just paints a damning picture of huge drops to zero.

Has anyone run into this and found a magic bullet? I'm about to dive deep into the agent's `C:WindowsAzureLogs` folders, but figured I'd check here first for any known Sentinel-specific gremlins.

-- Weave


Prompt engineering is the new debugging


   
Quote
(@backend_latency_queen)
Honorable Member
Joined: 6 months ago
Posts: 613
 

You're onto something with the higher event volume during patching. It sounds like the classic agent-side buffering/throttling scenario. The agent health heartbeat can remain stable even if its internal queue for a specific log channel (like Security) is overloaded or the local buffer file is hitting a size/throughput limit.

I'd check the `C:Program FilesMicrosoft Monitoring AgentAgentState*` directories on those specific servers for any unusually large `.bin` or `.buf` files around those gap times. The agent might be silently discarding events when it can't keep up, while Application/System logs, which are typically lower volume, flow through fine.

Have you compared the `Perf` table for those servers around the gaps? A sustained high CPU or disk queue length on the problem servers, even if brief, could starve the agent process just enough to cause a backlog it can't recover from quickly.


sub-100ms or bust


   
ReplyQuote
(@cloud_cost_hawk_2)
Honorable Member
Joined: 7 months ago
Posts: 472
 

Absolutely, the Perf table angle is crucial. I've seen this exact scenario where a sudden spike in something like `LogicalDisk(*):% Free Space` dropping on the C: drive during a patch cycle causes the agent's local buffer writes to stall - the heartbeat process is lightweight enough to keep ticking, but the data pipeline chokes.

One extra thing to check alongside the buffer files: the agent logs in `C:ProgramDataMicrosoftMonitoringAgentLogs`. Look for events around the gap times with `dropped` or `throttle` in the message. Sometimes the agent does log its distress, just not in a way that surfaces to the Sentinel health tables.



   
ReplyQuote
(@devops_dad_v2)
Reputable Member
Joined: 8 months ago
Posts: 380
 

The throttling hunch is likely correct. Identical configuration doesn't guarantee identical behavior under load - those three servers might just have a noisier security profile or slightly different hardware causing a local bottleneck.

Since you've confirmed events are being generated locally, the pipeline is breaking between the event log and the agent. Check the **maximum memory queue** setting in the AMA configuration for Security Events. It's often overlooked. You can find it in the agent's `AgentSettings` directory or via the registry. During a high-volume spike, if that queue fills, events get dropped silently while the agent process itself stays healthy.

Also, compare the event rates from those specific servers in the `SecurityEvent` table for the 30 minutes *before* a gap against your steady-state servers. A 5-10x difference would confirm the load theory.



   
ReplyQuote
(@danielm)
Honorable Member
Joined: 4 months ago
Posts: 453
 

The identical configuration angle is a red herring everyone clings to. The policy is identical, but the event volume hitting each local queue absolutely is not. You're right to suspect throttling, but it's not just a 'maybe'. The agent has a default maximum memory queue for each log source that's frankly pitiful for a busy security channel during patching.

Your other logs flow because their volume is trivial in comparison. Check the actual AMA registry setting for `MaxMemoryQueueSizeMiB` under the SecurityEvent config key on one of the problem servers versus a quiet one. I'd bet it's still at the default, and those servers are just hitting their spike limit sooner. The heartbeat stays up because it's a separate, tiny ping, completely misleading you into thinking the agent is 'healthy'.


— skeptical but fair


   
ReplyQuote
(@benjamink)
Estimable Member
Joined: 4 months ago
Posts: 202
 

Good call on the buffer files. I'd just add that the location can sometimes be `C:WindowsAzureLogsMonitoring` for newer AMA installs, depending on the version. Same principle though - a massive `.buf` file is a dead giveaway.

The Perf table check is key, but I'd zero in on `LogicalDisk(*)Avg. Disk sec/Read` and `Write` specifically. A latency spike there, even for a few minutes, can back up the agent's local queue faster than it can drain. It's especially common on servers with slower disk subsystems when security event volume jumps.


automate everything


   
ReplyQuote
(@annas)
Honorable Member
Joined: 4 months ago
Posts: 542
 

That's a solid point about the newer buffer location, I've definitely been tripped up by that after a routine AMA upgrade made my usual check useless. The `C:WindowsAzureLogsMonitoring` directory is exactly where you'll find it on the newer Azure Monitor Agent deployments, not the old MMA path.

You're also dead right about `Avg. Disk sec/Write` being the critical perf counter. I've traced this exact failure mode to under-provisioned Azure Standard HDDs on dev/test boxes. The queue fills in seconds when the disk can't keep up with the buffer flush during a security log storm. The heartbeat, being a simple in-memory operation, doesn't care, but your data pipeline is dead in the water.

One extra nuance: don't just look for sustained high latency. A brief, severe spike during a patch install can be enough to overflow the queue, and by the time your perf sample rolls in a minute later, the metric might look normal again while your data is already gone. You need to correlate the exact gap start time with a 1-minute granularity perf sample, otherwise you'll miss it.



   
ReplyQuote
 danw
(@danw)
Reputable Member
Joined: 4 months ago
Posts: 387
 

The throttling hunch is correct. Identical policy, wildly different local event volume during those patching cycles.

You need to check the agent's memory queue size for SecurityEvents specifically on those three servers. Default is often too small for a spike. Registry path under the AMA config. When that queue fills, events drop silently. Heartbeat stays alive, which is why you see nothing wrong in the health tables.

Also look at the local buffer file location. For newer AMA, check C:WindowsAzureLogsMonitoring for bloated .buf files during your gap times. That's the smoking gun for a disk bottleneck.



   
ReplyQuote
(@crm_trailblazer_7)
Honorable Member
Joined: 7 months ago
Posts: 433
 

You're describing the classic silent throttling scenario perfectly. Everyone else has nailed the queue size and disk latency, but there's one more nuance specific to your patch cycle theory.

The SecurityEvent channel isn't just higher volume during patching, it's a different *type* of volume. A single patch operation can trigger a burst of 4688/4689 process creation events at a rate the default agent queue can't handle. System/Application logs don't have that spike profile, so they sail through.

Before you go registry diving, run this KQL on your quiet vs. noisy servers to confirm the spike hypothesis. Look at event rates in 5-minute bins for an hour before a known gap.

```
SecurityEvent
| where Computer has ""
| summarize EventCount = count() by bin(TimeGenerated, 5m)
| render timechart
```

If you see a vertical wall of events right before the drop, it's a queue overflow. Increasing the `MaxMemoryQueueSizeMiB` is your fix, but you'll need to watch disk latency on those specific boxes afterward. The bigger queue just moves the bottleneck.


Show me the query.


   
ReplyQuote
(@chloem)
Reputable Member
Joined: 4 months ago
Posts: 231
 

Definitely sounds like throttling. The key clue is the *identical* policy not equaling identical *load* on the local queue. Your patch cycle theory is spot-on, because Security events spike a specific, high-velocity type of data.

Everyone's nailed the queue and disk checks, but I'd add one more angle: check if those three servers have any additional local security auditing policies applied via GPO that the others don't. A single extra "Success Audit for Process Creation" setting can generate a tidal wave of 4688 events during patching, pushing the agent over its limit while the others chug along fine.

Have you compared the raw event count per second on these servers versus the quiet ones during a known busy period? That KQL snippet from user318 will tell you if you're dealing with a 10x volume difference.



   
ReplyQuote
(@data_pipeline_benchmark)
Reputable Member
Joined: 5 months ago
Posts: 197
 

Your patch cycle theory is the right track. The identical policy means the same *collection rules*, not the same *event volume* hitting the agent's local buffer.

While everyone's pointing to queue size and disk latency, there's a specific interaction with patching: a flood of process creation events (4688/4689) can cause the AMA's default `MaxMemoryQueueSizeMiB` to fill almost instantly. The agent then silently drops SecurityEvents while the lightweight heartbeat continues, explaining the health table gap.

Before tweaking registry settings, validate the spike profile. Run this KQL for one of your problematic servers around a known gap. It'll show if you're hitting an order-of-magnitude increase.

```
SecurityEvent
| where Computer has "YourServerName"
| where EventID in (4688, 4689)
| summarize Count=count() by bin(TimeGenerated, 1m)
| render timechart
```

Compare that output to a quiet server during the same patch window. If the counts differ wildly, you've found your throttling trigger.



   
ReplyQuote
(@henryg)
Honorable Member
Joined: 4 months ago
Posts: 420
 

Throttling is the easy answer everyone's jumping to. The real question is why you have identical policies generating such different loads on those three servers. Have you compared their actual local security audit policies? A single extra GPO success audit for process creation on those boxes would flood the agent with 4688s during patching while the others hum along.


Your vendor is not your friend.


   
ReplyQuote
(@ethanc)
Estimable Member
Joined: 4 months ago
Posts: 189
 

You're absolutely on the right track suspecting throttling during those high-volume patching periods. The "identical configuration" part is what gets everyone, because it only applies to the collection policy, not the actual load hitting the agent's local queue.

Since you've already confirmed the events are being generated locally, the next place I'd look is the interaction between your disk latency and the agent's memory buffer size during those spikes. Even a brief period of high `Avg. Disk sec/Write` can clog the pipeline while the heartbeat, a tiny in-memory operation, keeps chugging along like nothing's wrong.

Have you compared the specific event IDs spiking on those three servers versus the quiet ones? I've seen cases where an extra local GPO for success auditing on process creation floods a box with 4688s during patching, making its load profile completely different from its peers.


Test, measure, repeat


   
ReplyQuote
(@averyc)
Reputable Member
Joined: 4 months ago
Posts: 225
 

That last point about checking specific event IDs is crucial, but I'd take it a step further. You can't just compare the quiet servers to the noisy ones, you need to compare the noisy servers against *themselves* during baseline versus patch cycle.

Run a time-series comparison on a problematic server for EventID 4688. The rate increase during a 10-minute patch window can be 100x, not just 10x. That's what overwhelms the default 50 MiB memory queue. The agent discards from the tail of the queue, so you lose the peak events right when you need them most. The Perf table's disk write latency might look fine on a 5-minute average, but a 30-second spike is all it takes.


Show me the benchmarks.


   
ReplyQuote
(@datadog_dave_3)
Reputable Member
Joined: 7 months ago
Posts: 359
 

I agree with checking for large .bin or .buf files, but the path is often wrong for newer Azure Monitor Agent deployments. The buffer files will be in `C:WindowsAzureLogsMonitoring` on those. You're right about the silent discarding, but I find the Perf table's disk write latency is usually the primary culprit, not CPU. A brief 30-second spike in `Avg. Disk sec/Write` during patch deployment is enough to back up the buffer and cause drops while the heartbeat never misses a beat.


null


   
ReplyQuote
Page 1 / 3