Showcase: Our alerting setup for when deliverability metrics dip.

Alright, gather 'round the campfire of cold, hard data. I see a lot of you getting excited about the latest "AI-powered deliverability insights" platform that costs more per month than your entire email infrastructure. Meanwhile, your actual inbox placement is tanking because you're staring at pretty dashboards instead of building systems that scream at you when things go sideways.

Let's talk about alerting on deliverability metrics without selling a kidney to a SaaS vendor. The goal is simple: when key metrics dip—hard bounce rates, spam complaints, a sudden drop in open rates from a major ISP—we know *before* the sales team starts panicking. This isn't about watching a graph; it's about actionable, noisy alerts that force someone to look at the problem *now*.

Our setup is deliberately boring and built from components we already run for everything else. We treat email metrics as just another time-series data stream. Here's the gist:

1. **Data Collection:** We use a lightweight exporter we wrote that polls our ESP's APIs (SendGrid, Postmark, whatever) every 5 minutes. It fetches the metrics we care about: `bounce_rate`, `complaint_rate`, `opens_rate` (per sending domain, per IP pool). It exposes them in a simple Prometheus format.
2. **The Critical Part – Baselines:** You can't just alert on a static threshold like "complaint rate > 0.1%". A campaign blast will look different from a transactional stream. We use recording rules in Prometheus to maintain a 7-day rolling average for each stream, and alert when the current value deviates by more than 150% from that baseline. This catches relative drops, which is what matters.

The alert rules themselves live in Prometheus and are routed via Alertmanager to Slack (for urgency) and a persistent PagerDuty incident if it's a critical, multi-hour trend. Here's a simplified example of one of our core rules:

```yaml
groups:
- name: deliverability_alerts
rules:
# Alert on complaint rate spike relative to its own baseline
- alert: ComplaintRateSpike
expr: |
(
espi_complaint_rate{stream="transactional"}
> on(stream) (7d_moving_avg_complaint_rate{stream="transactional"} * 1.5)
)
and
(espi_complaint_rate{stream="transactional"} > 0.08)
for: 15m
labels:
severity: page
category: deliverability
annotations:
summary: "Complaint rate spike for {{ $labels.stream }}"
description: "Current: {{ $value }}%. Baseline (7d avg): {{ $labels.baseline }}%. Check suppression lists and recent content."
```

The `and (espi_complaint_rate > 0.08)` is a sanity check so we don't page someone because a baseline of 0.01% jumped to 0.015%. It needs to be both a relative spike *and* absolutely bad.

We pair this with a separate, simpler system that scrapes Google Postmaster Tools data (via their underwhelming API) for IP/domain reputation and alerts on any drop below "Medium". That's a canary in the coal mine.

The whole thing runs in our existing Kubernetes cluster, costs virtually nothing beyond our time, and most importantly, it has no "magic." We know exactly why an alert fired, because we defined the logic. No black box "trust our algorithm" nonsense. When it breaks, we can fix it. Try getting that from your six-figure annual contract with the "deliverability intelligence" startup.

Now, tell me why I'm wrong and you need that shiny platform instead.

-- cynical ops