Your query highlights the fundamental issue with cross-platform comparison. The `email_delivered_total` metric's definition is entirely vendor-dependent. For a meaningful benchmark, you must first establish a ground truth by auditing each platform's contractual and technical definition of a 'delivered' event. The variance often exceeds 5% due to how they handle deferred bounces or mailbox provider filtering.
I built a similar comparison last year and found the only stable metric was the time delta between your application's 'handoff to vendor' timestamp and the recipient's first engagement (open/click). This bypasses the black box of vendor logging. Instrument your sending code to log a high-resolution timestamp the moment the API call returns a 2xx, then correlate it with the first engagement webhook.
Without this, you're not measuring platform performance; you're measuring their reporting consistency.
Data never lies.
That's a clever workaround, using the 'handoff to first engagement' delta. It cuts through a lot of the vendor definition noise.
Just be mindful that you're now measuring a different thing, and it can introduce its own bias. For a transactional email like a password reset, that delta is a solid performance metric. But for a marketing newsletter, the "first engagement" might be days later, and that delay has more to do with subscriber behavior than vendor speed.
So it's a great stable metric, but mainly for time-sensitive workflows. For batch sends, you might still need to wrestle with those delivered definitions.
You're right about the contracts. It's a step we often skip in the dashboards phase. I'd add that even after you pull the definitions, you have to track when they change. Most vendors will update their legal terms without a version bump in their API docs.
The "visualizing their marketing" line is spot on. I've seen teams make vendor decisions based on these kinds of charts, not realizing they're just seeing who has the loosest definition of success.
Review first, buy later.
Your query is flawed because you're dividing counts of apples by counts of oranges.
`email_delivered_total` is defined differently for each vendor. One might log it on SMTP acceptance, another on a delivery receipt from a receiving server. Your dashboard isn't showing performance, it's visualizing three different event definitions.
Look up the definitions in each vendor's API documentation and contracts. You'll likely find you can't compare these numbers directly at all.
Five nines? Prove it.