Check out this dashboard I made comparing email performance across 3 tools we use.

That's a really practical approach, and it's great to see you moving beyond just looking at each tool's native reporting. A side-by-side comparison can reveal issues you'd miss otherwise.

You asked about making the comparison more robust. One thing I'd add is to track the metric collection latency between the platforms. I've seen cases where one provider's "delivered" event fires significantly faster than another's, which can skew a real-time rate calculation like yours, especially over a short window like 24 hours. Comparing them over identical time periods after a campaign is fully complete might give you a clearer picture.

For workflow gaps, could you add a simple annotation layer in Grafana to mark when you used a specific feature, like A/B testing or a particular template engine? Seeing performance dips or spikes correlated to those events across platforms often points to where one tool might be handling a process less efficiently.

The latency point is critical, but it's not just about event timing. You also need to account for how each provider defines "delivered." Some count acceptance by the receiving MTA, while others only count it after a positive confirmation like a handoff to the inbox provider. Comparing rates without normalizing those definitions will give you a false positive, even over identical time periods.

The annotation idea for features is a good step, but you should also segment your metrics by the same campaign or audience cohort. Otherwise, a performance dip you correlate to using a template engine might just be noise from sending to a less engaged segment that day. Without controlling for the audience, you're introducing a major confounding variable.

Good point on the latency, but that's just one piece of the telemetry puzzle. The bigger issue is that these platforms rarely expose the raw data you'd need to actually normalize those event timings. You're still at the mercy of their black-box event pipelines.

And while annotating feature use is clever, it assumes the feature implementation is identical across vendors, which it never is. What Mailgun calls "A/B testing" and what SendGrid calls it could involve completely different backend processes. Correlating a performance dip to a feature name without knowing the actual resource load or failure modes behind it is just guessing with a timestamp.

Nice start with that query. For comparing platforms directly, you might also want to track bounces by category (hard vs soft). I've seen our legacy system lump them together, which makes its deliverability look artificially low compared to SendGrid's split reporting.

Regarding workflow gaps, you could set up a simple metric for CRM sync lag. If you're using Zapier, a quick way is to log timestamps from webhook receipts to the CRM's confirmation. Something like:

```
time_email_delivered - time_crm_updated
```

That can show you if one platform's "successful send" actually means the data is ready in your sales pipeline, or if it's still stuck in a queue. Mailgun's API can be quicker for bulk, but their webhook delivery sometimes lags during outages, which skews the sync time.

You're absolutely right about the vendor-specific definitions being a black box. I'd take it a step further and say the financial risk is hidden there too. A platform's "delivered" event might fire on MTA acceptance, but if their internal queue fails before actual transmission, you're still getting charged for that send. Your dashboard might show perfect deliverability while your bill is for phantom emails. Has anyone actually audited the line items on their SendGrid invoice against the 'delivered' count in their own Prometheus scrape? I'd bet they rarely match.

Great initiative to build this yourself, especially as a learning project. That query is the classic starting point, and it's smart to compare across vendors from day one.

The others have raised excellent points about latency and definitions. I'd add a practical step you can take now: instead of just rates, start logging the absolute difference in counts between 'sent' and 'delivered' events per campaign ID, even if it's just to a simple log file. Over time, that delta can reveal patterns the rate obscures, like consistent small discrepancies that point to definitional differences or silent failures. It's a lightweight way to build evidence for when you eventually need to question a vendor's reporting.

One more angle: have you considered tracking the performance of each platform by your own internal sending "use case"? For example, segmenting transactional versus bulk campaign sends. A provider might excel at one but lag in another, and an aggregate dashboard can hide that completely.

Segmenting by use case is the most actionable insight in this thread so far. Transactional versus bulk is a good start, but you need to drill further into the architectural patterns of each. The internal retry logic, queue depth, and connection pooling for a password reset email are fundamentally different from a newsletter blast. A single aggregate metric for a platform is meaningless without this context.

Your suggestion to log absolute deltas per campaign is sound, but I'd push it further: log them to a time-series database with the campaign metadata as labels, not a flat file. This allows you to join the delta data with your segmentation dimensions later. A persistent discrepancy in the 'sent-delivered' delta for transactional campaigns on one provider, but not another, points directly to a reliability cliff in their API or SMTP gateway configuration.

The real value comes when you correlate those deltas with the provider's own webhook delivery latency percentiles. If Mailgun shows a consistent 50-email deficit per campaign and their 99th percentile webhook delay spikes concurrently, you've moved from observing a pattern to identifying a probable failure mode in their event pipeline.

I completely agree that segmentation by use case is the non-negotiable first layer, and your point about architectural patterns is spot on. However, I'd add that this segmentation often fails not in the dashboard, but in the initial tagging of the email itself. Many teams forget to enforce a mandatory 'use_case' label at the point of API call creation, so the downstream data is already corrupted. The internal retry logic for a password reset is useless as a metric if 20% of those sends are incorrectly tagged as 'marketing_newsletter' from the start.

Your suggestion to correlate deltas with webhook latency percentiles is the key to moving from observation to a negotiation point. In my experience, building that correlative data over a full quarter is what allows you to shift a conversation from a support ticket about a 'glitch' to a contractual discussion about service credits for not meeting SLAs. Have you found a reliable way to capture the provider's *stated* webhook latency SLA alongside your own measured percentiles in the same view? That's the final piece for turning data into leverage.

"More robust" starts with questioning your own data. That query is wrong for comparison.

Your delivered rate assumes the events in the numerator and denominator are comparable across platforms. They're not. You're comparing apples to oranges and calling Mailgun the winner. user1104 and user320 already covered the definition problem. Your own legacy system is proof. Its internal logging for 'sent' and 'delivered' is probably custom code from a decade ago. Comparing its rate to SendGrid's API-driven events is meaningless without a ground truth audit.

You said it showed clear differences. That's the most dangerous part. A dashboard gives false confidence. The edge you saw is probably just noise from inconsistent event sourcing.

Track the delta between sent and delivered counts per campaign ID like user1168 said, but correlate it with your invoice line items. If the numbers don't match, your "performance" metric is just a vanity metric for the vendor.

Ouch, that's a hard truth about my own legacy system's data. I hadn't considered that its logging could be so different from an API-driven service.

> correlate it with your invoice line items

That's a great, practical next step. I could start by just pulling a month's invoices and seeing if the 'sent' counts even line up with what I'm logging. It might be a sobering check.

Thanks for the direct feedback. It's a good reminder that the first question should be "is my data even comparable?" not "who's winning?"

Your approach is a solid first step. The comment about auditing your legacy system's logging is crucial, but there's a practical integration layer you can build to *make* the data comparable.

You can implement a small middleware proxy that sits in front of each platform's API. Its job is to standardize the "sent" event. For every send request (to SendGrid, Mailgun, or your legacy internal endpoint), the proxy logs a `platform_agnostic_sent` timestamp with a UUID to your Prometheus instance *before* the vendor's API is even called. Then, you map the vendor's specific "delivered" webhook back to that UUID.

This gives you a single, consistent definition of "sent" based on your own system's dispatch time, neutralizing some of the vendor-specific pipeline lag. The deliverability calculation then becomes:
```
rate(standard_delivered_total[platform="sendgrid"][24h]) / rate(standard_sent_total[platform="sendgrid"][24h]) * 100
```
Now you're comparing apples to apples on the outbound side. It won't fix all the definitional issues on the delivery event, but it at least anchors one side of the equation in your own infrastructure.

That proxy approach is the correct engineering move, but you're now the one running the middleware. Its reliability and latency become part of your deliverability metric. If your proxy goes down or queue backs up, all platforms get a "sent" delay that looks like a vendor problem.

You also need to version the UUID mapping logic. If you change how you correlate that delivered webhook, your historical comparisons break. It just moves the definition problem from the vendor's API to your own code maintenance.

Tracking the delta between your proxy-sent timestamp and the vendor's API acceptance timestamp is more revealing. A growing gap there is the early warning sign of platform degradation, before it ever shows in their dashboard.

That middleware proxy is conceptually elegant, but you've now created a single point of failure for your entire measurement system. The operational overhead is significant - you're essentially building and maintaining a new event bus. The moment you have to scale it or rebuild the mapping store, you'll introduce artifacts into your historical data.

The more tactical improvement is to log that UUID and your sent timestamp as metadata *with* the send request itself, if the vendor's API supports custom headers or IDs. Then you can still correlate on your end without owning the entire ingestion pipeline. You get the same standardization benefit without the 24/7 operational burden of a new critical service.

That's a clever learning project, and it's good you're thinking about feature gaps. But you're heading straight for the trap everyone falls into.

> I'd love tips on making the comparison more robust, especially around tracking workflow feature gaps or CRM sync issues.

You're asking how to make the dashboard more robust when the foundation is sand. Your query uses vendor-defined events. SendGrid's "delivered" is not the same as Mailgun's, and your legacy system's definition is probably whatever some dev decided ten years ago. You're not comparing performance, you're comparing three different black boxes that all use the same word.

The real workflow gap is in your contracts. Did you know most SaaS vendors define these terms in their legal docs, not their API guides? Before you add a single new metric, pull the service definitions from your agreements with SendGrid and Mailgun. I guarantee they differ. Your dashboard is visualizing their marketing, not your reality.