Sorry to jump in late, but this is super timely for me. You mentioned alerting on all failures for a specific component. Are you filtering those alerts at all within PagerDuty, or does every single failure trigger a new incident there? I'm worried about getting flooded if something starts failing repeatedly before we can fix it.
Good foundation, but your Alert's filtering logic is incomplete. The `eventSources` filter you started only scopes the *type* of event, not the *severity*. You'll get notified for every `info` event from those sources, which defeats the purpose of a FinOps alert.
Add a `eventSeverity` filter set to `error`. Your manifest should look something like this:
```yaml
spec:
eventSources:
- kind: HelmRelease
name: "spot-autoscaler"
eventSeverity: error
summary: "FinOps Automation Failure: HelmRelease {{ .event.involvedObject.kind }}/{{ .event.involvedObject.name }}"
```
Also, the secret creation command you shared is missing the routing key value. It should be `--from-literal=address=`. An empty `address=` will cause a silent authentication failure.
The core setup is correct, but your secret creation command is fatally incomplete. The `--from-literal=address=` argument is missing the actual routing key value. It should be `--from-literal=address=`. As others noted, an empty address leads to silent auth failures. More critically, your approach of filtering on a specific HelmRelease `name` is brittle and won't scale. You'll need a separate alert for every cost-related component across all environments.
Use label selectors in your `eventSources` filter instead. If you label your HelmReleases with something like `finops-component: true`, you can filter generically and capture failures from your spot autoscaler, GPU node group scaler, and any new components you add later.
```yaml
spec:
eventSources:
- kind: HelmRelease
labelSelector:
matchLabels:
finops-component: "true"
eventSeverity: error
```
This also addresses user576's point about severity, which is essential for avoiding noise. Without the `eventSeverity: error` filter, a successful reconciliation (`info` event) after a fix would also trigger a PagerDuty event.