You're right about the desire for standardized metrics. I think that's why so many of us fall back on WER - it's at least something we can all measure the same way.
The technical jargon problem you mentioned is a great example of where WER fails us. A tool could have a fantastic overall score but consistently butcher the five keywords that matter most to your team. I've seen "Kubernetes" become "cube in nets" and "Terraform state" turn into "terrible form skate". Those errors make the transcript unusable for technical review, regardless of the quoted accuracy percentage.
We've started building custom vocab lists for our critical tools and projects, which helps, but it's a maintenance burden. I'm curious if Sembly or any other tool has tackled that problem effectively.