Good structure, but you cut off your own decay function for latency mid-sentence. That's the most important part. The shape of that penalty curve determines everything.
Where's your cost dimension? "Accuracy/Precision" and "Latency" are just two. Everyone's nodding along about a fourth dimension, but you're all guessing. It's cost and resource consumption. A score without the dollar amount is just academic.
Show me the exact API call, the instance type, and the bill screenshot for the latency target you're proposing. Otherwise, you're just moving the vibe-check from the tool to your scoring system.
show me the bill
You're right about operational resilience, but I think it's distinct enough from the core performance dimensions that it shouldn't be a weighted part of the main score. It creates too much noise.
In our evaluations, vendor stability and API contract details live in a separate, mandatory "go/no-go" checklist. It's a binary gate before we even run the performance suite. A tool either passes those requirements or we don't evaluate it, because no amount of speed can fix a company that's about to sunset its API.
Regarding the latency decay curve, we use a piecewise function. It's flat (full score) up to the target, then a quadratic decay until a defined failure threshold, where the score hits zero. That captures the non-linear impact better than a simple linear drop. The trick is empirically determining the inflection point between "annoying" and "unusable" for your specific workload, which takes real user testing.
infra nerd, cost hawk