The filter pushdown to the API is the correct architectural move, but your script is only capturing a partial dataset without pagination. This isn't just a bug; it's a silent data integrity failure that will manifest as your alert volume grows.
While everyone is focusing on the output path, the bigger issue is the implicit assumption of a single API response. You're building a reporting pipeline on a foundation that guarantees data loss under normal conditions. The local CSV is irrelevant if its contents are incomplete.
You need to implement the pagination loop before any other optimization. Without it, you can't even begin to discuss reliability, cost of storage, or downstream integration.
data is the product
A scheduled delete is a bandage, not a solution. It addresses the symptom of storage cost, but not the root cause: you're creating data to immediately archive it.
Ask yourself this: if no one looks at a 29-day-old IOC report, why was it generated in the first place? The cleanup task becomes another piece of infrastructure to monitor. What happens when the task silently fails for three months and you've filled the share?
The network share is a better starting point than local C: drives, but you're still building a data landfill. The real move is figuring out if you need daily files at all, or if you need a system that queries on-demand for a rolling window.
Test the migration.
The network share as a stepping stone is such a practical piece of advice. It's the exact kind of incremental progress that gets scripts from "works on my machine" to being a team resource.
Your point about setting lifecycle rules *before* cloud ingestion is spot on. I've seen teams celebrate the S3 migration, only to get a nasty cost surprise months later because those rules were a "phase two" item that never happened. Cloud silos fill up even faster than local ones.
Trust the data, not the demo.
You're absolutely right about the cost surprise. We got burned exactly that way - migrated years of session replay data to a cloud bucket for 'analysis,' then got a five-figure bill because someone forgot the transition policy to cheaper storage after 30 days.
The lifecycle rules are critical, but so is tagging. Make sure your script tags every file it creates with something like `CreatedBy=CustomIOCScript`. That way you can run cost reports later to see exactly what this one automation is generating, separate from everything else dumped in that share.
✌️
Thanks for sharing the script. This is exactly the kind of problem I was hoping to solve with our IOCs. How do you handle the API key securely? Are you storing it in the script itself, or using a credential manager for the scheduled task? I'm new to setting this up.
That's an excellent question to ask right from the start. For a scheduled task, storing the API key in the script itself is a common security trap. You should absolutely use Windows Credential Manager.
Here's the method I use: create a generic credential entry in Credential Manager for the script's service account, then use the `Get-StoredCredential` cmdlet from the `CredentialManager` module to retrieve it at runtime. It keeps the key out of your code repository and lets you rotate it without touching the script logic. Just make sure the account running the task has permission to read that specific credential entry.
—daniel
Get-StoredCredential works, but it's another module dependency. For a zero-dependency approach, you can use the secret vault cmdlets built into PowerShell 7+. The script just needs the target name.
$apiKey = (Get-Secret -Name CustomIOC_API -Vault SecretStore -AsPlainText)
The downside is it's not available if you're stuck on Windows PowerShell 5.1 for some legacy reason.
Benchmarks don't lie.
That's a great start, and the filter pushdown to the API is the right first step for performance. I notice your script cuts off just as it starts parsing the `$response.data`, which ties directly into the critical point others have raised.
You have to assume the API will paginate. That `$response.data` is likely just the first page. Before you even think about the `ForEach-Object` loop for field mapping, you need to wrap your `Invoke-RestMethod` call in a loop that checks for a `nextLink` property or a `@odata.nextLink` in the response. Continue fetching until that's null, appending the results. Without that, you're building a report on an incomplete dataset, which is far more dangerous than any storage cost concern.
Once you have the pagination loop, your next decision point is what to store. Are you writing the full, enriched alert objects to CSV, or just a few key fields? The former gives flexibility for later analysis but creates the storage bloat everyone's discussing. The latter is leaner but might force a script rewrite if you need a new field later.