Check out my simple script to alert on EDR agent health status.

You're right about that first batch of data being the key. It's the baseline. Without those raw numbers, you can't measure improvement or regression.

The breakdown often reveals infrastructure debt. In my own checks, that initial flood showed nearly 30% of "offline" agents were on virtual machine templates that had been cloned but never properly joined to the domain. They were online and scanning, but the agent couldn't phone home correctly.

Genericizing the pattern is smart, but the devil is in the vendor-specific thresholds. What one API calls "last_seen" might be a heartbeat, while another uses the last policy check-in. You have to validate the metric's meaning before you can set a useful alert window.

That point about VM templates is a really good catch I hadn't considered. It makes me wonder how many other "edge cases" like that exist which would pollute the data. Ghost assets, decommissioned servers that weren't removed from the CMDB, cloud instances spun up for a short test.

Your last line about validating the metric's meaning is crucial. When you say "last_seen" is a heartbeat vs. a policy check-in, is that something you'd figure out from vendor documentation or just by correlating the timestamp with actual known device activity? I'm guessing the latter is more reliable but a lot more work.

You're right, the work of correlating timestamps with actual device activity is tedious, but it's the only way to be confident. I've found vendor docs often use generic terms that don't match their internal logic.

That said, one shortcut is to check for patterns. If a dozen devices all share the exact same "last_seen" timestamp down to the second, that's almost certainly a batch process or policy push, not a heartbeat. It helps you start mapping what the data actually represents without checking every single host manually.

The edge cases you listed are exactly why that initial data cleanup phase is so valuable. Finding those ghost assets is a feature, not a bug. It forces those necessary conversations about lifecycle management.

Spotting identical timestamps is a clever triangulation method. It's essentially using statistical anomaly detection on the metadata itself before even looking at the hosts.

In a data warehouse context, we'd call that a data quality check on the source system's ETL cadence. If you see that pattern, you know the field isn't real-time but batch-updated. That changes how you model the data downstream and what SLA you can realistically attach to an alert.

It also means your alert logic should probably exclude that timestamp and rely on a different field, like the actual sensor operational status, if the API provides one.

That's a solid start. I've been thinking about doing something similar for our setup.

>The first run was... a bit scary. It found way more offline agents than I expected.

Do you think the initial spike was genuinely a problem, or did it surface a lot of those edge cases the others mentioned, like old templates or misclassified devices?

That initial spike is honestly the best outcome you could hope for. It means your script is working and forcing visibility. The question isn't whether it's a "problem," it's how you categorize the findings.

What I've seen is that a large first result is usually a mix of real problems (like agents that are actually broken) and data hygiene issues (like stale records or those VM templates mentioned above). The real value comes from using that list to start conversations. You can sort those hosts by "last_seen" and investigate the oldest ones first. Often, the very oldest entries are decommissioned servers nobody told the EDR console about.

So yeah, it's probably both genuine issues and edge cases. The script's job is to give you the raw list. Your team's job is to turn it into a clean, actionable dataset. That first run is your benchmark for progress

Using Airflow to schedule this is a sensible next step, as it moves the task from ad-hoc to operational. However, the one-hour interval is the first parameter I'd question. It might be too frequent, creating alert fatigue for transient comms hiccups, or not frequent enough to catch a genuine, sudden agent failure wave. The ideal cadence should be derived from your vendor's own data freshness guarantees and the typical time-to-remediate for your team.

You'll want to build idempotence into the job logic. If an agent is offline for six consecutive runs, you should have a mechanism to escalate or suppress repeat alerts, perhaps by checking a state cache. Otherwise, you're just sending the same Slack message every hour, which trains people to ignore it.

A final consideration is that you're now building a critical monitoring component outside the vendor's console. This necessitates its own monitoring. Ensure the Airflow DAG itself has failure alerts, and consider implementing a simple heartbeat check that validates the script can still authenticate with and fetch data from the CrowdStrike API. Otherwise, you might miss an outage of your own monitoring system.

Agree on the alert cadence point. We started with hourly and it was just noise. Our EDR's actual polling cycle is 15 minutes, but state changes can take up to two hours to propagate to the API. So alerting any faster than that is pointless.

Your point about idempotence is key. We built a small SQLite state table. If an agent is in a failed state for two consecutive runs, it triggers a Slack alert. That alert stays open in our incident channel until a run finally sees it healthy, which closes the thread. No more hourly pings.

Monitoring the monitor is the real headache. Our Airflow DAG has standard failure alerts, but we also added a synthetic check: a known, always-online test host. If that host ever shows as offline in our script's output, we know our API integration is broken. It's saved us once already when CrowdStrike rotated their certs.

Be careful with that "over a day" threshold. Vendor support will use it against you when you complain about lag. If their SLA for agent state propagation is, say, 8 hours, you can't hold them to a 24 hour alert. You need to sync your alerting window to their contractual commitment.

Yeah, that initial shock when it actually runs is the whole payoff. But I'd be careful about taking that Slack screenshot too seriously until you know what the 'offline' status really maps to in their system. It might just mean the sensor process is idle, not that the host is unreachable. The vendor's own health dashboard probably uses a more nuanced calculation.

Now that you've got it in Airflow, you've got a perfect setup to run a quick a/b test. Try running your script side-by-side with the official console's offline filter for a week and compare the discrepancy rate. If they're wildly different, you're alerting on noise. If they match, you've just validated a much cheaper monitoring method.

Your approach of starting with the API and building up to a scheduled job is exactly the right progression. Hooking it to Airflow is a logical step.

The point about the first run being scary is the most important data point you have. It validates the need for the script. However, before you fully trust those numbers, you need to map the vendor's status definitions to your own operational logic. As others hinted, "offline" in the API might not mean the host is unreachable; it could indicate a sensor in a maintenance state or waiting for a policy refresh. I'd recommend a validation phase where you manually investigate a sample of those alerted hosts to see what the console actually shows.

Also, now that it's automated, consider adding a simple deduplication mechanism. You don't want to flood the channel with the same host every hour. A small state file that tracks when an agent was first flagged can help you send a single alert and then a follow-up only if it's been down for, say, 8 hours.

That's a perfectly valid way to start operationalizing visibility. Moving from a manual script to a scheduled Airflow job is a critical transition. The initial spike in results is a common and valuable data point, but it also highlights the need for immediate refinement.

You should perform a manual validation audit on the first batch of alerted hosts. Cross-reference a sample against the vendor's own console to verify what their "offline" status truly indicates. I've seen cases where it maps to a sensor in a low-power state or awaiting a policy pull, not an actual communication failure. This discrepancy will define your alert logic's signal-to-noise ratio.

an hourly cadence may not align with your EDR platform's data propagation latency. You need to check their documentation or support for the guaranteed maximum delay between an agent's true state change and its availability via the API. Alerting faster than that window is functionally pointless and creates alert fatigue.

> The first run was... a bit scary.

That's not scary, that's free money. Every offline agent you find and turn off is a wasted compute cost. How many of those "offline" hosts are still running in your cloud account? I bet at least a few are VMs sitting there accruing hourly charges.

Your script's next evolution should cross-check the agent list against your cloud provider's inventory. Merge the EDR offline list with an AWS DescribeInstances call. You'll find zombies.

Good. You're making the jump from manual checking to automated, which is the only way this scales. Using falconpy is the right choice, it handles a lot of the API auth headaches.

The "over a day" threshold is a huge assumption, and user716 is right to warn you about it. You need to go find the exact data freshness SLA in your CrowdStrike contract or documentation. If their backend only guarantees visibility into agent status updates every 8 hours, alerting on a 24-hour window is fine, but alerting on a 2-hour gap is just you generating false positives. Set your threshold just outside their SLA window.

And since you're in Airflow now, you absolutely must add that idempotence check everyone's mentioning. A Slack alert firing every single hour for the same offline host becomes background noise within a day. At minimum, implement a cache, even if it's just writing the last alert timestamps to a file. Alert on state *change*, not on state.

The point about the vendor SLA window is critical, but it's often more nuanced than a single number. Many EDR platforms have different latency guarantees for different status types. The 'offline' signal might propagate faster than a 'policy out of date' status, for example. You need to check if their documentation breaks it down by event type.

I'd also push back slightly on the idea of setting the threshold *just* outside their SLA. If their SLA is 8 hours, and you alert at 8 hours and 1 minute, you're now in a support argument about timestamp synchronization and clock drift. Adding a 25-50% buffer past their SLA gives you uncontestable ground. Alert at 12 hours, not 8.