I've been running Mandiant Threat Intel feeds into our SIEM for about six months now, primarily for enrichment and to add context to our internal alerts. Over the past two weeks, I've noticed a persistent and troubling pattern: a significant portion of the Indicators of Compromise (IOCs) delivered in the daily feed appear to be exact duplicates from the previous week's batches. This is creating noise and bloating our indicator database.
My initial assumption was a parsing or ingestion error on our end, but after a thorough review of our log pipelines and the raw feed files, the duplication is present in the source data from Mandiant. To validate, I performed a hash comparison on the `.json` files delivered for the `malware_families` and `threat_actor_indicators` feeds over the last 14 days. Here's a simplified snippet of the script I used to check for duplicate `mscore` values and indicator IDs within a date range:
```python
import json
from collections import defaultdict
def find_duplicate_iocs(file_paths):
seen_ids = defaultdict(list)
for file_path in file_paths:
with open(file_path, 'r') as f:
data = json.load(f)
for indicator in data.get('indicators', []):
ioc_id = indicator.get('id')
seen_ids[ioc_id].append(file_path)
return {iid: paths for iid, paths in seen_ids.items() if len(paths) > 1}
# Example output showed the same indicator ID appearing in files from 2024-03-10 and 2024-03-17
```
The results confirmed that numerous indicators, across multiple object types (file hashes, domains, IPs), are being re-delivered with identical `last_updated` timestamps or with timestamps that are only a day apart, not reflecting any new intelligence.
This behavior has several operational impacts:
* It inflates our storage and indexing costs in Splunk unnecessarily.
* It causes alert fatigue for our analysts, as our correlation rules trigger on "new" indicator matches that were already processed and deemed non-actionable the prior week.
* It complicates compliance reporting for controls like PCI DSS 11.4 and HIPAA, where we must demonstrate timely ingestion of *new* threat intelligence.
My questions for the community are:
* Has anyone else observed this specific issue with feed duplication in Mandiant Threat Intelligence, particularly within the last month?
* Is this a known characteristic of their delivery methodology—perhaps a "rolling baseline" where a certain percentage of indicators are re-sent for confirmation—that I've misunderstood? I've reviewed their documentation but found no mention of this behavior.
* What has been your successful mitigation strategy? Are you performing deduplication on your end based on the Mandiant `id` field and the `last_updated` timestamp, or have you engaged with support to resolve it at the source?
I am currently implementing a temporary deduplication filter in our ingestion pipeline, but I'd prefer to receive clean feeds. Any insights into your workflows or communications with Mandiant on data hygiene would be greatly appreciated.
Logs don't lie.