Check out what I made: A Prometheus rule generator that writes back to Git

Hey folks! I've been wrestling with Prometheus alerting rules in our GitOps flow for months. We'd update a rule in a PR, merge it, but then the actual alert would fire hours later when ArgoCD finally synced. 😩 I wanted something that could react *immediately* to metric changes.

So I built a little operator that bridges the gap! It watches a specific Prometheus `rules.yml` file in a Git repo, and when a new alert condition is detected (like a service's error rate spiking), it **generates a new, temporary alert rule on the fly and writes it back to a branch** for review. This gives us instant visibility while still keeping the Git record.

Here's the core idea in a nutshell:

1. **Watches** Prometheus for a set of "template" alert conditions (e.g., `job:error_rate > 0.05`).
2. When triggered, it **generates a concrete rule** with specific labels and annotations, targeting the offending service.
3. It **applies this rule directly** to the Prometheus server via its API for immediate effect.
4. Simultaneously, it **commits the new rule** to a new Git branch (like `alert-hotfix/service-x-`) and opens a PR.

Here's a snippet of the rule generation logic (Python with `kubernetes` and `prometheus-api-client`):

```python
def create_rule_from_template(alert_name, observed_labels):
# Start with our template rule for 'HighErrorRate'
base_rule = yaml.safe_load("""
- alert: HighErrorRate
expr: job:error_rate{service="SERVICE_PLACEHOLDER"} > 0.05
for: 2m
labels:
severity: page
scope: generated
annotations:
summary: "High error rate for {{ $labels.service }}"
runbook: "https://our-runbook/error-rate"
""")
# Replace the placeholder with the actual service from the firing alert
base_rule[0]['expr'] = base_rule[0]['expr'].replace(
"SERVICE_PLACEHOLDER",
observed_labels['service']
)
# Add the specific instance to annotations
base_rule[0]['annotations']['instance'] = observed_labels['instance']
return base_rule
```

The Git write-back uses a simple GitPython workflow. The real magic is having the operator run in the cluster with a GitHub token (from a Secret) so it can push branches.

**Why I like this approach:**
* **Speed:** We get alerts within seconds, not hours.
* **Audit Trail:** Every auto-generated rule is still code-reviewed via PR. No "shadow" rules.
* **Cleanup:** The PR merge (or close) is the trigger for the operator to remove the temporary rule.

Has anyone else tried something similar? I'm curious about alternative patterns for dynamic alerting within a GitOps framework. The main challenge I'm tackling next is pruning stale generated rules.

-- Weave