Announcing the Community Benchmarking Project - join the working group

db_diver · 2026-08-06T00:10:57Z

The perennial challenge in our field is the gap between vendor-provided performance data and real-world, apples-to-apples workload comparisons. We often discuss the merits of Amazon Aurora's storage layer versus Google Cloud SQL for PostgreSQL's machine types, or the true cost-performance ratio of managed services versus self-hosted alternatives on equivalent hardware. These discussions are frequently hampered by a lack of a consistent, transparent, and reproducible benchmarking framework. Therefore, I am proposing the formation of a Community Benchmarking Project working group. The objective is to collaboratively design, develop, and execute a suite of standardized benchmarks that reflect complex, modern use cases. This goes far beyond simplistic synthetic queries. We aim to model: * Mixed OLTP/analytical workloads with varying read/write ratios. * Connection pooling and concurrency under load, simulating application burst behavior. * The impact of specific features like Aurora's global database, Cloud SQL's high availability configurations, or Spanner's interleaved tables. * Operational considerations such as failover time, backup performance, and point-in-time recovery granularity. The initial phase will focus on a core set of managed relational and hybrid systems: **Amazon RDS & Aurora (PostgreSQL/MySQL), Google Cloud SQL & Spanner, and Azure Database for PostgreSQL**. The intent is to later expand to include managed NoSQL offerings like Memorystore (Redis), Azure Cosmos DB, and Amazon Keyspaces (Cassandra). I envision the working group will need to tackle several key technical components: * **Infrastructure-as-Code templates** (Terraform/Pulumi) for provisioning identical, transient benchmarking environments across clouds. * A **standardized schema and data generation tool** capable of producing a realistic, skewed dataset of configurable scale (e.g., 100GB to 10TB). * A **workload driver** (perhaps extending tools like pgbench, sysbench, or Yahoo! Cloud Serving Benchmark) to simulate application logic with parameterized queries. A preliminary sketch of a potential benchmark driver configuration might look like this, defining a complex transaction: ```yaml workload_profile: "ecommerce_mix" transactions: - name: "user_checkout" weight: 0.7 # 70% of transaction mix steps: - "SELECT * FROM user_cart WHERE user_id = ? FOR UPDATE;" - "UPDATE inventory SET stock = stock - ? WHERE sku = ?;" - "INSERT INTO orders (...) VALUES (...);" - "DELETE FROM user_cart WHERE user_id = ? AND sku = ?;" - name: "analytics_dashboard" weight: 0.3 # 30% of transaction mix steps: - "SELECT category, SUM(revenue) FROM orders WHERE order_date > NOW() - INTERVAL '7 days' GROUP BY category;" ``` I am seeking members with deep operational experience in these platforms, particularly those who have conducted performance tuning or comparative analysis. Contributions can range from designing representative data models and queries, to writing provisioning code, to executing test runs and analyzing results. The ultimate deliverable will be a public repository containing all tooling, configurations, and a regularly updated set of results published under a Creative Commons license. This will serve as a community resource to ground our discussions in empirical data.

Page 2 / 2 Prev

Announcements

Last Post by finops_auditor_ray 55 minutes ago

16 Posts

16 Users

0 Reactions

1 Views

RSS

finops_auditor_ray

(@finops_auditor_ray)

Reputable Member

Joined: 4 months ago

Posts: 240

06/08/2026 7:21 pm

You're setting a massive scope there. Before anyone even thinks about replicating Aurora's storage layer, you need to define the exact cost profile of the "equivalent hardware." That's the first trap.

I've seen too many comparisons where someone runs a benchmark on a self-hosted node, forgets to factor in the three-year reserved instance cost for the compute, the provisioned IOPS, and the cross-AZ networking for high availability, and then declares a 60% savings. The numbers are meaningless without the full bill.

Where's the working group's plan for capturing and verifying the complete, all-in TCO for each test configuration? If that's not step one, the entire project is academic.

show me the bill

ReplyQuote

Page 2 / 2 Prev