We're stuck in the "spreadsheet and email" dark ages for sharing Indicators of Compromise. Our SOC team identifies malicious IPs and domains, but getting them to the network and cloud teams is a manual, error-prone mess. We don't have budget for a dedicated Threat Intelligence Platform (TIP) or a fancy SOAR.
I need a method that's:
* Machine-readable (so network gear can ingest it)
* Has some basic context (like threat type and confidence)
* Doesn't require a dedicated server or paid service
* Allows for versioning or at least knowing what's current
I've considered:
* A shared Git repo with flat files (STIX/TAXII seems overkill for this).
* A simple REST API off an internal server we already have.
* Pushing JSON blobs to an S3 bucket or internal file share.
What's the simplest, most robust pattern you've actually seen work? Show me the actual workflow or a config snippet. If you suggest a tool, I need to know the exact setup steps, not just a name.
For example, is something like this sustainable for a few hundred IOCs per week?
```json
{
"generated": "2023-10-26T15:00:00Z",
"iocs": [
{
"value": "203.0.113.45",
"type": "ipv4",
"threat": "c2_server",
"confidence": "high",
"first_seen": "2023-10-25",
"comment": "From campaign XYZ"
},
{
"value": "malicious.example.com",
"type": "domain",
"threat": "phishing",
"confidence": "medium",
"first_seen": "2023-10-26"
}
]
}
```
How do you handle updates, deletions, and false positives? How do the consuming teams (e.g., network security with their firewalls) actually pull and apply this data?
Show me the query.
I'm an infrastructure architect at a fintech company with about 600 people. We run a hybrid cloud setup split between AWS and on-prem data centers, and I've been responsible for plumbing threat intel from our security team into our WAFs, network ACLs, and cloud security groups for the last three years. Our current system handles about a thousand IOC updates daily.
Core breakdown of the patterns you're considering:
1. **Shared Git Repo with Flat Files**
* **Integration Effort:** Almost zero for the SOC team. They just commit a JSON file. The real work is on the consuming side: every network device, cloud account, or SIEM needs a client script to pull and parse. You'll end up writing and maintaining 5-10 small cron jobs or Lambda functions.
* **Hard Limitation:** No push mechanism. Your firewall won't know an IOC is updated until its next scheduled pull. For a few hundred IOCs a week, a 15-minute poll is fine. For real-time blocking needs, it's useless.
* **Hidden Cost:** The "free" part is a trap. You're trading license costs for developer hours to build and, more importantly, maintain the glue code. When GitLab changes its API or your SSH keys rotate, everything breaks at once.
* **Where It Wins:** For pure, auditable versioning and historical tracking. Every change is a commit with a hash and an author. You can't beat it for knowing *exactly* what was blocked and when.
2. **Simple REST API on an Existing Server**
* **Real Load:** A basic Flask/FastAPI app behind nginx can easily handle 50-100 RPS on a $10/month VM. Your constraint won't be performance; it'll be authentication and access control.
* **Deployment Gotcha:** You now own a *critical security service* on a server that probably has other duties. A patching reboot takes down IOC distribution for all teams. You must implement rate limiting and logging, or a buggy script from the cloud team will DOS the endpoint.
* **Honest Workflow:** The SOC team needs to update the backing datastore (a Postgres table or even a SQLite file). They'll want a UI, which means you're suddenly building a mini-TIP. This path has the highest scope creep.
* **Specific Setup:** If you go this route, don't build a generic API. Build one specific endpoint: `GET /iocs/active.json?since=ISO8601`. Return exactly the JSON structure your consumers need. No query flexibility, no GraphQL.
3. **JSON Blobs in S3/File Share**
* **Real-World Throughput:** S3 can serve a 100KB IOC file to thousands of consumers concurrently with no performance tuning. A Windows file share will choke after a few dozen simultaneous connections.
* **Cost:** S3 costs are negligible ($0.023 per GB storage, pennies in requests). The real price is in egress if your consumers are in a different cloud region or on-prem ($0.09 per GB). Keep your payloads lean.
* **Versioning Pattern:** Don't rely on S3 versioning. Include a `generated` timestamp in the filename: `iocs-2023-10-26T15-00-00Z.json`. Your consumers always pull the latest by pattern. Keep a `iocs-latest.json` symlink (use an S3 copy operation) for ease.
* **Breakage Point:** Permissions and networking. If your S3 bucket isn't accessible from your on-prem network or the cloud team's VPC, the system is dead. You'll spend more time on IAM roles and VPC endpoints than on the actual solution.
4. **STIX/TAXII on a Shoestring**
* **Your Instinct is Right:** It's overkill for a few hundred simple IOCs. The complexity isn't in the format; it's in standing up a TAXII server (`medallion` or `opentaxii`) and making every consumer speak the protocol. It's a multi-week integration project per consumer.
* **When It's Actually Simple:** If, and only if, *all* your consuming tools already have native TAXII client support (some SIEMs and firewalls do). Then it's just another feed to add. If they don't, you're back to writing clients.
My pick is **Option 3: S3 bucket with timestamped JSON files**, because your primary goal is getting machine-readable data from point A to points B, C, and D with minimal ongoing ops. The workflow is concrete: your SOC team runs a script that validates their internal data, generates a JSON file with the `generated` timestamp, uploads it to S3 with the timestamped name, and updates a `latest.json` pointer. Every consumer has a scheduled task that pulls `latest.json`, compares its `generated` field to its last pull, and processes if newer. I'd only switch to the Git repo pattern if you have a hard requirement for full immutable history and your team lacks AWS access.
keep it simple
I've actually run a variant of that S3 bucket pattern for about two years now, and it worked surprisingly well for a similar scale. The key was using S3 event notifications as a poor man's pub/sub system to trigger consumers.
Our SOC would drop a new JSON file like the one you sketched into a bucket. A simple Lambda attached to the `PutObject` event would then fan out notifications to SNS topics - one for network team scripts, one for the cloud accounts. Each consumer team just had to subscribe their own script to their topic. This avoided the "poll and parse" cron job mess.
The real benefit wasn't the storage, it was the decoupling. The SOC never needed to know who the consumers were or if their scripts were down. The versioning was just the timestamp in the filename. The one caveat: you absolutely need a dead-letter queue on that SNS setup to catch any failing subscribers, otherwise you'll silently lose updates.
~jason
That JSON structure you've got looks solid for a few hundred IOCs. The key is sticking to it rigidly so your parsers don't break. We used almost the same format, but we added a mandatory "ttl_hours" field for automatic cleanup in firewalls. It stopped us from accidentally blocking an IP forever if the feed went stale.
The real simplicity came from pairing it with the S3 method user775 mentioned. We had our SOC script dump that exact JSON to a bucket, and the filename included the epoch timestamp `iocs_1698332400.json`. Every consumer just checked that number against their last pull - no need for complex diff logic, they just grabbed the newest file. It's dead simple and the versioning is built in.
My only warning is to test your parsers with a malformed file early on. Have your SOC team commit a test JSON with a missing bracket once. You'll find out quickly which team's script crashes and burns.
Keep automating!