Tagging is the correct approach, but the specific format of that tag is crucial for parsing downstream. You mentioned tags like `titan-gateway-security-detailed`. I'd recommend adding a consistent key-value structure, such as `profile=security-detailed`. This makes it trivial to write a single, reusable extraction rule in your SIEM, rather than a new regex for every arbitrary tag name.
A potential caveat is that some older SIEM parsers can choke on whitespace in the tag field. If your tag contains a hyphen, it's usually safe, but test with a single gateway first to confirm your syslog daemon doesn't add extra brackets or alter the format.
Also, remember that indexing rules based on these tags are only cost-effective if your SIEM applies them *before* indexing. If it indexes first and filters later, you're still paying for the storage.
prove it with data
Great point about the key-value structure for tags. That saved me a ton of time when we had to scale up to dozens of profiles across different environments. We landed on `env=prod:profile=security-detailed` as a format.
Your warning about indexing rules is spot on and easily overlooked. Our first Splunk bill had a nasty surprise because the heavy parsing was happening post-index. We had to rework the entire props.conf setup to handle the tags at ingest, otherwise you're just paying to store noise. It's a backend config that's easy to miss if you're focused on the gateway side.
The `env=prod:profile=security-detailed` format is excellent for large-scale management. It forces a taxonomy from the start.
One technical nuance we found: if you're using a syslog relay like rsyslog or syslog-ng before your SIEM, you must verify the relay passes the structured tag intact. Some relay configurations, especially those that normalize priority or timestamp, will strip the tag or re-encapsulate the message, breaking your parsing. Always test by sending a sample and inspecting the raw message arriving at the final SIEM ingestion point.
Your props.conf comment is critical. It's not just about cost, but also search performance. If your tags aren't parsed at index time, every query scanning those logs has to run the extraction regex, which slows down your dashboards and alerts.
Mike
The CLI is for viewing, not for setting up external streaming. You can't configure it there.
All external log streaming is controlled by the Director's Log-Streaming Service. You define profiles (like 'detailed-flows' or 'security-events') and attach them to a collector, which is just a syslog destination (host:port). You then bind that collector to the gateway device.
To get the data you want, you'll need at least two profiles on the Director: one with log level 'Informational' for flow data, and one with 'Detailed' for security events with context. Bind them to the same collector, but use distinct tags as discussed.
Data over opinions