Hi everyone,
I’m currently helping my team evaluate a few Claw family agent platforms for automating some of our internal reporting workflows. We’ve gotten demos from three vendors, and now I’ve been tasked with creating a weighted scorecard to help us make a data-driven decision.
I have a basic list of criteria (like cost, SQL support, ease of integration, quality of documentation), but I’m a bit stuck on how to structure the scoring in a fair and consistent way. Specifically:
* How do you decide on the weights for each category? Is there a common framework?
* How granular should the scoring be (e.g., 1-5 scale, 1-10)?
* Do you score each feature as just present/absent, or on how well it’s implemented?
I started a simple table in Excel, but I’m wondering if there’s a better way. I was thinking of using something like this in SQL later to aggregate scores from different evaluators, so a structured approach would be great.
```sql
-- Rough idea for a scoring table structure
CREATE TABLE vendor_scores (
evaluator_id INT,
vendor_name VARCHAR(50),
category VARCHAR(50),
score DECIMAL(3,1),
weight DECIMAL(3,2)
);
```
Has anyone built one of these for agent evaluations before? I’d love to see any templates or real examples of rubrics you’ve actually used. What categories did you find most important for Claw agents specifically? Any pitfalls to avoid? 😅
Thanks in advance for any guidance!