So our security team finally decided they wanted "better visibility" into the Firebox logs. Naturally, they asked if we could just pipe everything into the existing Grafana/Prometheus/Loki stack. "Sure," I said, already knowing the WatchGuard ecosystem's usual... *charm*.
Turns out, the path of least resistance isn't the Firebox's built-in Syslog-to-Loki magic (because that doesn't exist), but a two-step shuffle:
1. Point the Firebox Syslog at a reliable forwarder (we used `rsyslog` on a small VM).
2. Have that forwarder ship structured logs to Loki via the Promtail agent.
The configs aren't wild, but you have to massage the syslog format into something Loki can label properly. Here's the key bit of the `promtail` config that handles the Firebox's syslog output:
```yaml
scrape_configs:
- job_name: watchguard_firebox
static_configs:
- targets:
- localhost
labels:
job: firebox_syslog
device: firebox_m500
__path__: /var/log/firebox/**/*.log
syslog:
listen_address: 0.0.0.0:8514
idle_timeout: 60s
label_structured_data: yes
labels:
facility: "{{.SYSLOG_FACILITY}}"
hostname: "{{.SYSLOG_HOSTNAME}}"
source: "{{.SYSLOG_MSG}}"
```
The gotchas? Oh, there are always gotchas:
* The Firebox's syslog timestamp format *mostly* plays nice, but you'll want to check for timezone drift if you're aggregating across regions.
* Traffic logs are verbose. Set up some log line filtering in `rsyslog` before they hit Promtail, or you'll burn through Loki storage on the noise.
* The `SYSLOG_HOSTNAME` label is gold—lets you slice dashboards per device when you have more than one box.
End result? We got pretty graphs for threat detection events and VPN user activity right next to our app metrics. Took about half a day to get it reliable. The security folks are happy, and I got another benchmark for "log integration latency."
benchmarks or bust
Your two-step approach is correct for WatchGuard devices. I've found the syslog output from Fireboxes to be surprisingly stable, but the volume can spike during DPI operations. That `idle_timeout: 60s` is wise; we had to increase ours to prevent connection churn under load.
One caveat on labeling: make sure you parse the structured data for the event type. We extracted the `firebox_event_type` field with a regex stage in the promtail pipeline *before* the syslog stage, which let us create more useful log streams than just by facility and hostname. It cut our query latency significantly because we could target specific event streams directly.
Did you run into any issues with the timestamp format? We had to write a custom timestamp stage because the Firebox was sending microseconds, which Loki's default parser rejected.
--perf