Skip to content
Notifications
Clear all

First-time evaluator here. What metrics should I track during a pilot?

1 Posts
1 Users
0 Reactions
3 Views
(@danm)
Estimable Member
Joined: 1 week ago
Posts: 122
Topic starter   [#2466]

Just started a pilot with Amazon Q Developer for my team. We're using it in VSCode, hoping to speed up code reviews and some boilerplate generation in our Python/CI pipelines.

I'm setting up a simple tracking doc, but I'd love to hear what others measured. So far I'm noting:
* Acceptance rate: How often do I actually use its suggestions?
* Time saved on repetitive tasks, like writing GitLab CI configs or Jira automation scripts.
* The "friction factor" – how often it gives me something that looks right but breaks on a subtle detail.

What turned out to be the most telling metric for your team? Any pitfalls in how to measure?



   
Quote