I've been observing a persistent and disruptive pattern in our ETL environment since migrating several data pipeline orchestrators to Azure VMs protected by Microsoft Defender for Endpoint. We rely heavily on scheduled PowerShell scripts for tasks ranging from file system hygiene, to API data extraction, to triggering downstream BigQuery jobs via `bq` command-line tool. Over the last quarter, we have documented 17 instances of legitimate, internally-developed scripts being abruptly quarantined, causing pipeline failures and requiring manual intervention to restore.
The quarantines appear stochastic, not deterministic. A script will execute successfully for dozens of cycles before being flagged. The commonalities we've identified so far are:
* Scripts that dynamically build and execute `Invoke-RestMethod` or `Invoke-WebRequest` commands with parameters assembled from configuration files.
* Scripts that generate and execute other PowerShell code blocks (e.g., for templated database deployments).
* Heavy use of `System.IO.Compression.FileSystem` or `Expand-Archive` on downloaded data payloads.
Crucially, the scripts are all signed with our internal code signing certificate, and they operate within the principle of least privilege. The Defender console typically shows a generic "Behavior:Win32/Powershell.ABC!ml" or "Trojan:Script/Foreign.A" alert with high confidence but scant technical details.
A representative example of a script that was quarantined mid-day after weeks of stable operation:
```powershell
# Script to fetch and unzip daily transaction batch for processing
$dateStamp = (Get-Date).ToString("yyyyMMdd")
$sourceUri = "https://internal-api.example.com/export/$dateStamp.zip"
$localZipPath = "D:StagingInput$dateStamp.zip"
$outputPath = "D:StagingUnzipped$dateStamp"
# Download the batch file (quarantine often occurs during or after this call)
Invoke-WebRequest -Uri $sourceUri -OutFile $localZipPath -UseDefaultCredentials
# Extract the contents - this is also a frequent trigger point
Expand-Archive -Path $localZipPath -DestinationPath $outputPath -Force
# Generate a manifest file for the downstream loader
$fileList = Get-ChildItem -Path $outputPath -Recurse -File | Select-Object FullName
$fileList | Export-Csv -Path "$outputPathmanifest.csv" -NoTypeInformation
```
Our current workaround involves adding the entire script directory path to the exclusion list, which feels like a significant security regression. I am seeking to understand:
1. Is this a known issue with Defender's AMSI integration or its machine learning model for PowerShell script behavior?
2. What specific script patterns or API calls are most likely to trigger false positives in a heavily automated enterprise environment?
3. Beyond exclusions, what is the recommended approach for stabilizing an automated data pipeline under Defender? Are there specific code signing practices or script invocation methods (e.g., using `-ExecutionPolicy Bypass` with specific flags) that improve its accuracy?
The operational cost of these random failures is mounting, both in SRE time and in missed SLAs for data freshness. Any detailed insights or benchmark experiences from other engineering teams would be invaluable.
--DC
data is the product
Signing helps, but Defender's behavior-based detection often kicks in after the fact. Your triggers are classic heuristic matches: dynamic code generation and network activity post-download look like script injection or payload staging.
Your real cost is the manual intervention 17 times. Quantify that. If each failure takes a team member 30 minutes to restore, that's over 8 hours of engineer time wasted last quarter. That's a measurable TCO hit on your VMs.
You need to feed these incidents back into Defender. Submit the quarantined scripts as false positives via the security center. If they're internal and signed, you can also create an exclusion path or process-based rule, though that weakens the posture.
cost per transaction is the only metric
That's a really thorough breakdown of the trigger patterns. The dynamic code generation one especially hits home.
I've run into similar issues with signed scripts in Salesforce data sync processes. Even with signing, if the script writes temporary files or modifies its own execution path, Defender's heuristics can still get twitchy.
Did you find a pattern with the time of day or VM load when the quarantines happened? I'm wondering if there's a correlation with when Defender updates its heuristics.
Yeah, quantifying the manual intervention time is the perfect way to frame it for management. It's not just an "annoying security quirk" - it's a real operational expense.
Submitting false positives is key, but in my experience, the turnaround can be slow. We've had more immediate success by creating a very narrow file hash allowance for our core, signed scheduler scripts. It doesn't solve the underlying heuristic issue, but it stops the production fires.
Have you found the exclusion rules to be stable? We've had a few get mysteriously wiped after major Defender updates.
✌️
Totally agree that quantifying the manual toil is the best way to get traction on this. We did the same math and it got us a dedicated review with the security team.
One caveat on the false positive submissions, though: in our case, Defender sometimes flagged the script *and* the resulting process tree. Submitting the script hash cleared the file, but we'd get hit again because the child process activity (like a spawned `bq` command) was still suspicious. We had to submit multiple layers to finally get it to stick.
The dynamic construction of `Invoke-RestMethod` is almost certainly a primary trigger. Defender's AMSI scans script blocks at runtime, and building command strings from variables defeats static signature analysis. It then flags the behavior as obfuscation.
You mentioned your scripts are signed, but that only validates the file on disk. The runtime behavior of assembling and executing code from external configs is treated as a new, unsigned script block.
Have you tested running the same logic from a compiled PowerShell module (.dll or .psm1) instead of a .ps1 script? The heuristics for compiled modules can be different, and it might bypass the specific detection for dynamic script generation.
BenchMark