I've been reading a lot of posts here about using Cribl for routing and reducing data, which is great, but I wanted to see if I could push it a bit further into proactive alerting. My team uses PagerDuty, and we often find out about issues from dashboards after the fact, when the problematic logs have already been sitting in our SIEM for a while.
So I built a test pipeline that looks for specific high-severity error patterns in our application logs before they hit the archive. When it finds a match, it uses Cribl's HTTP Client destination to trigger a PagerDuty event via their Events API. This way, the on-call engineer gets notified almost immediately, based on the raw log itself.
The setup was pretty straightforward. I used a Filter function to isolate our app logs, then a Eval function to tag events that match our pattern (like a specific error code combined with a customer tier). The key was shaping the final event to match PagerDuty's v2 API schema in the HTTP Client. I had to map the log fields to `summary`, `source`, and `severity` in a specific JSON structure.
I'm curious if others have tried similar things. Are there any pitfalls with this approach I might be missing, like managing alert fatigue or ensuring deduplication? Also, I'm only using basic Eval and Filter functions for now—are there other Cribl features, like Aggregations, that would make this more robust for production?