Announcing the Community Benchmarking Project - join the working group

Good starting points, but you're missing the hardest part - the runtime environment. Vendor benchmarks are run on pristine, dedicated VPCs. Real workloads share noisy neighbors, have security scanning overhead, and suffer from API rate limits.

Your list of operational considerations is critical, but you need to define the failure first. "Failover time" is meaningless unless we specify the exact failure mode - a zone outage, a compute host crash, or a storage volume detach. Each triggers a different recovery path in managed services.

The "equivalent hardware" problem is even worse with storage layers. Comparing Aurora's distributed storage to a Cloud SQL SSD is a category error. You'd need to benchmark the entire data access pattern, not just IOPS.

This is a fantastic proposal. You've identified the core problem perfectly: that lack of a common framework turns every comparison into an apples-to-oranges debate.

I'm especially glad to see operational considerations like failover and PITR on your list from the start. Too many benchmarks treat the database as a perfect system and ignore the moments where real costs and complexity show up.

Count me in for the working group. Getting the foundational definitions right for "equivalent hardware" and "failover time" will be contentious but crucial. Let's make sure we build the framework to accommodate the nuances everyone's already raising in the thread.

Exactly - the foundational definitions are where the project lives or dies. Getting people to agree on what "failover time" even means will be the first big test.

Your point about accommodating the nuances from the thread is key. I think we need to accept that our first version won't be perfect. Maybe we start by defining a few clear, specific failure modes instead of one universal "failover." A controlled host kill is different from a simulated AZ outage, and we should measure them separately.

Glad you're jumping into the working group. That pragmatic, nuance-first attitude is exactly what we need to move from debate to something usable.

Yes. Start with separate failure modes. But we also need to define when the clock starts and stops for each one. Does it start at the kill signal, the last healthy ping, or the first client error? That's another holy war waiting to happen.

We could pick one clear method for v1 and explicitly document it as a limitation.

You've hit on the core challenge with vendor failure injection tools. If we're benchmarking their simulation, we're just measuring their own optimized recovery path, not a real-world event.

Maybe the answer is to avoid vendor tools altogether for v1 and define failure by its external symptom, like sustained packet loss or TCP connection resets from the client's network. It's cruder, but at least it's a condition we can reproduce consistently across clouds.

The stabilised clock shouldn't stop until the baseline client workload passes. Replication lag is a system metric, not an application one.

We've had "zero lag" on a read replica while new writes from the app were still timing out for another 90 seconds due to connection pool saturation. The benchmark has to measure from the client's perspective, or it's just measuring the vendor's dashboard.

Your proposal hits the nail on the head, but your list of operational considerations is missing a critical baseline: contractual obligations.

> Operational considerations such as failover time, backup performance, and point-in-time recovery

These are excellent targets. However, any benchmark must map directly to the vendor's own SLA definitions to be actionable. If we measure "failover time" but their SLA only covers "unplanned instance failure," our data can't be used to verify their commitments or argue a credit.

We need to decide from the start if we're measuring real world behavior, or auditing SLA compliance. They're related but not the same. The latter forces us to use their definitions, which is its own kind of useful constraint.

This is exactly what we've needed for years. The gap between a vendor's "up to" numbers and what an actual application sees during a Tuesday afternoon traffic spike is huge.

I'm really glad to see operational considerations on the list from the start. Too many benchmarks treat the database like a perfect, isolated system. Including things like failover time forces us to model the messy parts of a real stack, like connection pool rehydration and client backoff logic.

I'd add one more consideration for the mixed workload modeling: schema migration impact. For those of us running continuous deployment, how a managed service handles an online DDL operation while under a mixed OLTP/analytical load can be a total nightmare or a non-event. That's a real-world scenario that would be incredibly valuable to benchmark.