So, I'm a data analyst who recently got pulled into helping our security team with some dashboarding for their new SIEM. My background is all SQL and Tableau, so I figured, "How hard can it be?" 😅
Well, I just got a major reality check. Our cloud bill for the SIEM's data ingestion and storage jumped by like 300% last month. Turns out, my "help" in setting up some new log sources (and not filtering *anything*) was a huge part of the problem. We never really planned a log retention policy or talked about cost control from the start.
In my data world, you usually keep everything because storage is cheap, right? But with SIEM logs, the volume is insane. I was just writing queries to pull data without thinking. For example, I set up a dashboard that needed Windows event logs, and I was pulling all EventID levels for every server. Looking back, I should have started with a much more targeted approach.
```sql
-- My naive 'get everything' query approach
SELECT *
FROM windows_events
WHERE timestamp > NOW() - INTERVAL '7 days';
```
Now I'm scrambling to learn about log filtering, deduplication, and tiered retention (hot/warm/cold). Has anyone else made this switch from analytics to security data? What are the biggest mindset changes or technical steps you took to get costs under control? Specifically:
* How do you decide what logs to filter out at ingestion vs. just not querying?
* Are there rules of thumb for retention periods for different log types (e.g., auth logs vs. application logs)?
* Any good ways to estimate costs before turning on a new log source?
Feeling a bit out of my depth, but trying to learn! The security team is nice, but they're also new to the cloud side of this.