Everyone talks about ingesting indicators, but the real mess is cleaning them up. ThreatConnect's built-in expiration is... optimistic. It assumes you trust the source's TTL, which is a joke.
What are you actually doing to automate purges? Custom scripts hitting the API? Hoping the Playbooks app doesn't break? Or just accepting a bloated, noisy indicator database?
Curious if anyone has a simple, maintainable approach that doesn't require another SaaS subscription or a full-time engineer to babysit.
Your stack is too complicated.
Totally feel your pain on the source TTL trust issue, it's more of a suggestion than a rule. We got sick of the bloat too.
We run a simple scheduled job on a lightweight VM that hits the TC API nightly. It looks for indicators past their expiration date *and* checks if they've been tagged in any active investigation in the last 90 days (we tag them when used). If not, they get purged. It's not fancy, but it's been solid for over a year now.
Honestly, maintaining that one Python script is way less hassle than managing Playbooks for this, in my experience.
That tagging approach is smart - gives you a safety net before deletion. We do something similar but also cross-reference our SIEM to see if any alerts fired on the indicator recently. If they did, we extend the TTL automatically.
Our script runs weekly, not nightly. Found that daily was overkill for our volume and just added API load. The weekly cleanup keeps things tidy without the overhead.
measure twice, ship once
Totally agree, trusting the source TTL is a dangerous game 😅. I'm still figuring this out myself.
That built-in expiration has burned us before, letting old junk stick around. We ended up writing a tiny cron job that uses the API to delete anything older than, say, 90 days, unless it has a specific "keep" tag. It's dumb but it works.
Can I ask, when you say custom scripts, are you using the TC REST API directly? I'm trying to set something similar up and the docs are a bit of a maze.
Containers are magic, but I want to know how the magic works.
Optimistic is putting it kindly. It's a design flaw to treat source TTL as anything but a suggestion, especially when half your feeds are repackaged from other feeds.
The "simple, maintainable approach" is a myth if you're talking about real security value, not just disk space. You can write a cron job to delete old entries, everyone does. But unless that logic includes a check against your actual investigation workbench and recent incidents, you're just automating the creation of false negatives.
We tag indicators linked to cases as "active". The purge script respects that. It's the bare minimum, not a solution. The noise isn't the problem, the missed signal is.
Trust but verify