Too many benchmarks just report average scores and declare a winner. Statistical significance is what separates a real improvement from noise. Here's the pragmatic approach.
For a typical accuracy/score comparison between two models on the same test set, use a paired test. The observations are paired by test item. McNemar's test is good for binary outcomes (correct/incorrect). For continuous scores, use the paired t-test if scores are roughly normal, or the Wilcoxon signed-rank test if they aren't.
Example with Python statsmodels for McNemar:
```python
import pandas as pd
from statsmodels.stats.contingency_tables import mcnemar
# Contingency table: [ModelA_Correct & ModelB_Correct, ModelA_Correct & ModelB_Wrong]
# [ModelA_Wrong & ModelB_Correct, ModelA_Wrong & ModelB_Wrong]
table = [[45, 15],
[7, 33]]
result = mcnemar(pd.DataFrame(table), exact=False)
print(f'Statistic={result.statistic:.3f}, p-value={result.pvalue:.4f}')
```
Key points: Ensure your test set is independent of training data. Account for multiple comparisons if you're testing across many datasets. A p-value < 0.05 is standard, but consider the effect size. A tiny significant improvement might not be operationally meaningful.
mw
Infrastructure is code.