I've been conducting a deep-dive evaluation of Grafana OnCall for a potential multi-team rollout, focusing heavily on its ability to reduce alert noise through intelligent suppression. A core requirement is the suppression of non-critical alerts during planned maintenance windows or known high-traffic periods. While the `time_period` field in the `oncall_schedule` resource seems designed for this, I've encountered persistent and counterintuitive behavior that undermines its reliability.
My configuration follows the documented pattern: I've created a schedule specifically for suppression, linking it to integration routes via the `slack` and `important` fields. The schedule is of type `web`, as we're managing this via Terraform. The issue is that alerts routed to this schedule are **not** being consistently suppressed during the defined "off" periods. They continue to page the on-call engineer, effectively bypassing the intended quiet hours. This creates a critical trust deficit in the tool.
Here is a representative Terraform snippet of the suppression schedule:
```hcl
resource "grafana_oncall_schedule" "maintenance_suppression" {
name = "Maintenance Suppression Window"
type = "web"
time_zone = "America/New_York"
shifts = ["maintenance_shift"]
time_period {
start = "2024-10-26T02:00:00"
end = "2024-10-26T06:00:00"
}
}
resource "grafana_oncall_shift" "maintenance_shift" {
name = "Maintenance Block"
schedule_id = grafana_oncall_schedule.maintenance_suppression.id
start = "2024-10-26T02:00:00"
duration = 14400 # 4 hours in seconds
users = [grafana_oncall_user.dummy_user.id] # Placeholder user
}
resource "grafana_oncall_integration" "prometheus_alerts" {
name = "Production Prometheus"
type = "grafana_alerting"
}
resource "grafana_oncall_route" "suppress_low_severity" {
integration_id = grafana_oncall_integration.prometheus_alerts.id
routing_regex = "severity=warning"
routing_type = "regex"
slack {
channel_id = "C1234567890"
}
important = false
# Intended link to suppression schedule
schedules = [grafana_oncall_schedule.maintenance_suppression.id]
}
```
**Observed Failure Modes & Investigation:**
1. **Time Zone Mismatch Hypothesis:** Verified that the `time_zone` on the schedule, the OnCall organization settings, and the source alert timestamps are synchronized. This does not appear to be the root cause.
2. **Schedule Activation State:** The schedule shows as "active" in the UI during its defined period, yet alerts are not being intercepted.
3. **Interaction with `important` flag:** Experimented with both `important = false` and `important = true` on the route, theorizing that "important" alerts might ignore suppression. The behavior was inconsistent; some suppressed, others did not.
4. **Direct vs. Web Schedule:** The documentation is ambiguous on whether `web` schedules are fully supported for suppression purposes, or if an `ical` type is required for robust time-based logic.
My questions for the community are architectural and practical:
* Has anyone successfully implemented reliable time-based suppression in Grafana OnCall, and if so, what is the precise configuration pattern?
* Are we misinterpreting the function of the `schedules` field on a route? Does it merely define *who* is on call during an alert, rather than *if* the alert should fire based on that schedule's active periods?
* Is the intended design that one must create an "on-call person" for the suppression window (e.g., a dummy user or a dedicated role) and rely on that person having no notification policies? This seems like an unnecessary workaround.
* What is the actual hierarchy of override? Does `important = true` trump schedule suppression?
The lack of clarity here pushes us towards implementing suppression at the alert source (e.g., in Alertmanager or Grafana Alerting), which defeats the purpose of a centralized on-call management layer. I'm seeking a definitive pattern that ensures operational integrity.
--from the trenches
infrastructure is code