Hey folks,
I've been neck-deep in evaluating our threat intel stack lately, and it got me thinking: we're paying a not-insignificant amount for several commercial intelligence feeds, but how do we *really* know we're getting our money's worth? It's easy to get dazzled by volume—"look at all these IOCs we're ingesting!"—but volume without quality is just noise that slows down your analysts and clogs your pipelines.
For me, coming from a DevOps/automation background, I want to measure this like we measure everything else: with data and clear metrics. It can't just be a gut feeling. So, I've been trying to build a bit of a framework to assess feed quality, and I'd love to compare notes with how others are tackling this.
Here’s what I’ve been tracking and some of the rough scripts I use to quantify it:
**Key Metrics I'm Measuring:**
* **Precision & Recall (But for Intel):**
* **False Positive Rate:** How many indicators from a feed, when deployed in our blocking controls (like network policies or EDR), cause alerts on legitimate business activity? I log this automatically.
* **True Positive Rate:** When we get a *confirmed* internal security incident, how often was an indicator from the feed already present in our system prior to the incident? This is the golden ticket, but harder to track.
* **Timeliness & Freshness:**
* **Mean Time to Indicator (MTTI):** The delta between when an indicator is first seen "in the wild" (per other sources) and when it appears in our paid feed. This requires correlating with open-source feeds.
* **Indicator "Age" at Ingestion:** Simple script to check the first-seen date (if provided) versus when we pulled it. A feed giving us stuff that's 30 days old isn't as valuable.
* **Operational Relevance & Context:**
* **Actionability Score:** A subjective but important rating from our analysts. Does the feed entry come with enough context (TTPs, campaign info, mitigation steps) to actually *do* something, or is it just a bare IOC?
* **Integration Ease:** How cleanly does the feed integrate into our TIP (ThreatConnect, in this case) and our downstream CI/CD for security automation? Are the APIs stable and well-documented?
**A tiny example of how I might log a basic freshness check for a feed file:**
```bash
#!/bin/bash
# Assumes feed provides a 'first_seen' timestamp in its JSON.
FEED_URL="https://feed.provider.com/v2/iocs.json"
DOWNLOAD_PATH="./daily_feed.json"
curl -s -H "Authorization: Bearer $API_KEY" "$FEED_URL" -o "$DOWNLOAD_PATH"
# Use jq to calculate average age of indicators at time of pull
AVG_AGE=$(jq '[.indicators[] | (.timestamp - .first_seen)] | add / length' "$DOWNLOAD_PATH")
echo "Feed ingested at $(date -Iseconds)"
echo "Average indicator age in feed: $(($AVG_AGE / 86400)) days"
```
This is just a start. I'm also curious about:
* **Coverage vs. Your Threat Profile:** Does the feed specialize in your industry vertical or the specific threat actors you're most concerned about?
* **Signal-to-Noise Ratio:** Pure volume is useless. What percentage of the feed's indicators are unique and not just duplicates of what you can get from reputable open-source projects like Abuse.ch?
So, how are you all doing this? Are you just trusting your vendor's reports, or have you built internal dashboards to track these things? Any particular metrics you've found to be the best "value for money" indicators?
Looking forward to the discussion.
bw
Automate all the things.