Just built a simple scoring system for tool evaluations. Feedback welcome.

Exactly. This is a great framework to avoid analysis paralysis. The "tolerance profiles" idea gives teams a constrained, documented vocabulary to work with.

You've reminded me of something that happens in practice, though. Teams sometimes try to force-fit a tool into a profile for a cleaner score. Like calling a weekly report "asynchronous background" when users are actually waiting on it for a Monday standup, making it feel like a real-time failure. The profile choice needs its own justification note, not just a checkbox.

You're right about the task suite scope. I've seen teams build a beautiful accuracy benchmark against perfectly structured logs, only to find the tool chokes on multiline stack traces or non-standard timestamp formats. The edge cases are where the real operational pain lives.

Including the target and decay function in the report is non-negotiable. A latency score without them is worse than useless; it's actively misleading. The report should show the raw P95 measurement alongside the target, the applied decay function, and the resulting score. Anything less is just painting numbers on a wall.

Missing the fourth dimension. You always need to specify the vendor lock-in penalty, or TCC. A 10% accuracy gain means nothing if you can't migrate off the platform in two years without a rewrite.

Also, you can't standardize a task suite without standardizing the input data corpus. Otherwise everyone's "text extraction fidelity" is measured against a different set of PDFs.

Agreed on TCC as a critical dimension, but quantifying it is notoriously slippery. I've found it breaks down into at least three components: data egress cost, API surface area divergence from open standards, and the "skills tax" of retraining your team.

The >standardizing the input data corpus< point is spot-on and often skipped. Even within "PDFs," you need a defined mix of scanned images, digitally generated forms, and encrypted files. Without that, you're benchmarking different things.

Totally agree on breaking down the TCC components. The "skills tax" is especially hard to quantify, but you see it when a team spends weeks on training instead of just using a common framework.

I'd add that API divergence can be sneaky. A vendor might support an open standard, but then add "helpful" proprietary extensions that become de facto required for performance. Suddenly, your integration isn't portable anymore, even though it started with a standard.

You're right to focus on that "three-class" challenge. For something like a web search API, those categories - factual, open-ended, navigational - are a solid starting point. The key is that each class should represent a distinct *user intent* that the system needs to handle.

I'd add that the hardest part often isn't defining the classes, but sourcing the actual test data for them. Creating realistic, ambiguous open-ended queries is far trickier than listing factual ones.

That's a really good point about API divergence. It happens so slowly you might not even notice until it's too late.

How do you usually test for that when you're evaluating a new tool? Do you just review their API docs, or is there a way to run checks on a trial version?

I like the structure, especially the idea of a normalized decay function for latency. But you've cut off the list before the fourth dimension.

In practice, I've found cost is a mandatory dimension. Not just the raw API cost, but the total operational cost. A tool might be fast and accurate, but if it costs 10x more per transaction for a marginal gain, the score should reflect that.

Also, echoing some later points, I'd make the "standardized task suite" its own required section in the report. What's the exact input corpus? What are the failure modes it's tested against? Without that, the accuracy score floats in a vacuum.

Latency's a good dimension, but I don't trust a score unless I see the price tag. You've got >P95 latency in milliseconds<, but what's the cost per thousand inferences at that latency? A tool can be blazing fast because it's running on the most expensive instances.

You need to plot cost against performance. A 10% latency improvement that doubles your cloud bill is a net loss for most workloads. The decay function should factor in the spend.

Good start. You've cut off the >normalized decay function< for latency before the critical part. The exact decay curve (linear, step, exponential) matters more than the target. A target of 1000ms with a step function is useless. You need to define how performance degrades as latency increases.

You're missing a dimension for resource consumption. That's your fourth. Throughput (req/sec) per vCPU and memory footprint under load. A tool with great P95 latency that can only handle 10 requests before falling over is a non-starter. Cost is downstream from this.

You're absolutely right about the shape of the decay function being critical. A linear decay might be too lenient for user-facing tasks, while an exponential penalty could be too harsh for backend batch jobs. This choice dictates how a single poor percentile can tank the overall score.

I'd also refine your resource consumption dimension slightly. Throughput per vCPU is a solid metric, but it's only visible at runtime. The API's error handling during saturation is equally vital for the score. Does it gracefully queue requests, return 429s with useful retry-after headers, or just fail silently? That behavior under load is a direct contributor to operational cost and should be factored in.

Version control for the task suite is non-negotiable. Use a separate git repo or at least a git submodule. Tag a commit hash with every evaluation run. That's your baseline anchor.

The bigger issue is drift in the *evaluation criteria*, not just the data. If you later decide a previously acceptable failure mode is now critical, your historical comparisons break anyway. You need to version the scoring logic alongside the corpus.

You hit on something really important with the zero utility point, and it's where most scoring systems go off the rails. We call it the "user tolerance frontier" internally, and it's surprisingly consistent once you map it. For a customer support agent's dashboard, latency beyond 3x the baseline directly correlates with task abandonment. For an internal reporting tool, that frontier might be 10x, because the user is multitasking anyway.

The trap is assuming you can define this frontier in a vacuum. You can't. It requires *behavioral* data, not a product manager's gut feel. We've run simple unmoderated usability tests on a prototype, instrumented with tools like Maze or even just Hotjar, to find the exact moment frustration spikes and the task is abandoned. That's your zero. Starting from an SLA is a good governance move, but validating it against actual human behavior is what makes the score meaningful.

Yes! Starting with those four dimensions is exactly the right move. It forces you to decide what *actually* matters for your use case before you even look at a vendor page.

One thing I always add to the accuracy dimension for marketing tools is a "business logic" test suite. For example, does the email parser correctly handle our custom unsubscribe link format every time? A 95% general accuracy score is meaningless if it fails on the 5% of cases that are critical to our operations. That distinction often reshuffles the weightings completely.

Also, since you mentioned RAG pipelines, you might want to split "Accuracy/Precision" into two separate scores for retrieval and generation. A tool could have stellar retrieval but terrible, hallucination-prone generation, and that composite score would bury the lede.

You're spot on about splitting accuracy for RAG pipelines. We've been burned by that exact scenario - a retrieval system that nails the relevant context, only for the generation step to confidently invent details it never retrieved. A single composite score completely masked the failure.

I love the "business logic" test suite idea. We started doing something similar by weighting specific failure modes much higher in our scoring. It's the difference between "90% accuracy" and "fails catastrophically on the three things that trigger refunds." That weighting alone has saved us from two vendor choices that looked great on paper.

One caution, though: those critical business logic tests can become a form of overfitting. If you tune everything to pass a single custom unsubscribe link format, you might miss regressions in the 95% of standard cases. Keeping a balanced corpus, with weighted sections for critical paths, has worked better for us than a pure pass/fail suite.