Hi everyone, I'm pretty new to the SIEM/SOAR side of things, coming from a marketing ops background where we dealt with unifying messy customer data. I guess the principles are similar, just different data sources!
I recently took on a project to reduce our SIEM ingestion costs and improve our ability to write consistent detection rules. We had firewall logs pouring in from six different vendors/models (Cisco ASA, Palo Alto, Fortinet, etc.), all in their own formats. It was a mess for our analysts.
I suggested we try Cribl Stream, since I'd heard good things about log normalization. My goal was to take those six formats and create a single, common schema before they hit our SIEM. Here’s a simplified breakdown of what I did:
First, I identified the common fields we absolutely needed for basic firewall analytics: source_ip, destination_ip, destination_port, action (allow/deny), and a normalized vendor/device identifier. The tricky part was mapping each vendor's unique field names and values to these common ones.
For example, Palo Alto uses `action="allow"` while Cisco ASA uses `action="permitted"`. I used Cribl's Pipelines to create a `normalized_action` field with just "allow" or "deny" for all logs. For Fortinet, the source IP was in a field called `srcip`, while Cisco used `src_ip`. I used the `Eval` function to handle these mappings, like: `if (vendor=='fortinet') source_ip = srcip`.
The real win was being able to drop a lot of redundant and verbose fields unique to each vendor *before* ingestion. I think we cut the volume of those logs by about 40% just by stripping out unneeded data.
I'm still learning, so I'd love to hear from others who've done this. How do you handle the ongoing maintenance when a vendor updates their log format? Do you create separate pipelines per source, or one big, conditional pipeline? Also, any tips on validating that the normalized logs are correct and haven't lost critical info?