Just had a real-world lesson I thought I should share. I was helping our security team investigate a potential phishing incident from a few months back. Went to pull the relevant email logs from Anomali... and they were gone.
Turns out, the default data retention settings for some log sources are surprisingly short. We never adjusted them after deployment. The data we needed had already been purged based on the default policy.
This seems like a major pitfall for anyone using it for compliance or forensic work. Has anyone else run into this? How are you managing retention timelines across different data sources? I'm curious what settings others have landed on.
And this is exactly why you never deploy a security tool without reading the data lifecycle config. It's not a shiny dashboard, it's a data pipeline.
The defaults are tuned for demo environments, not production. You got bit by the classic "set it and forget it" with a compliance tool. I've seen the same with ELK stacks and cloud log sinks.
Your security team should have a mandated retention period. Then you go set those numbers, not the other way around.
If it ain't broke, don't 'upgrade' it.
Ouch, that's a tough lesson to learn the hard way. I've seen this exact scenario play out, not just with Anomali but with other SIEM and log aggregation tools where the defaults are almost designed for a trial, not a real security operation.
One extra layer I'd add is that even after you set your retention periods, you need to reconcile them with your actual storage capacity. We set ours to a year for critical sources, but quickly realized the raw cost of that storage was prohibitive. We ended up tiering it: raw logs at 90 days, then key parsed fields and metadata stored much longer in a cheaper cold storage. It's a pain to manage, but it's the only way we could meet both our forensic and budgetary requirements.
What's your compliance driver? That's usually the key number. For PCI, it's 90 days of immediate availability. For us, a longer timeline for certain data is an internal infosec policy for investigating slow-burn campaigns.
customer first