Hey mods, thanks for the heads-up on this new rule. Makes total sense. Benchmarks without specs are just marketing fluff.
As someone who runs a lot of tool trials, I'll definitely include this info moving forward. Quick question: for cloud instances, do we need the exact machine type *and* the region, or is the machine type enough? For example:
* **Tool:** RenderForm
* **Test:** Batch generating 1000 personalized images
* **Instance:** AWS EC2 c5.2xlarge (us-east-1)
* **Result:** ~8.5 minutes
Is that the level of detail you're looking for?
Trial number 47 this year.
That's a really helpful example, thanks. I'm new to posting benchmarks and was wondering the same thing.
When you include the region, like us-east-1, does network latency or just the raw compute power vary enough to matter for, say, a local data processing test? I'm guessing it's best to list it anyway, just to be safe.
Your example is exactly right, especially for including the region. The instance type alone isn't sufficient for a proper benchmark. While the raw vCPU and memory specs of a c5.2xlarge are consistent across regions, the underlying hardware generation can differ. A c5.2xlarge in a newer region might be on newer Intel Cascade Lake versus older Skylake processors, which can impact performance for compute-heavy tasks like image rendering.
Also, including the region eliminates ambiguity about which specific SKU you used. Pricing and availability vary by region, so stating "us-east-1" allows others to accurately reproduce your test or understand the cost context. For full reproducibility, you might even consider noting if you used a dedicated instance or the default shared tenancy, as that can introduce some variability.
every dollar counts