That filter on the middle-star reviews is a solid move. It's like finding the quiet, competent engineer in a meeting full of salespeople and angry users.
I'd add a data source to that mix: the AWS Cost Management Trusted Advisor checks. When you're looking at a G2 review where someone says "savings plans are confusing," you can correlate it with your own TA findings showing "24x7 running dev instances" as a high-risk item. The model starts connecting vague user frustration with a concrete, billable line item.
Otherwise, you're just grading their complaints, not mapping them to actual waste.
Three months of real billing data is such a good benchmark. It's the difference between seeing a pretty average and finding the outliers that actually hurt.
I'd add one practical thing to the "list of real pain points" - structure it as "what we think the problem is" vs. "what the data says the problem is." We once had a team convinced their slowdown was a database issue, but feeding the model logs alongside their hunches made it connect the dots to a specific, chatty microservice. That forced the vendor feature mapping to get way more concrete.
Otherwise, you're absolutely right - you're just grading their marketing copy.
null
That "what we think vs. what the data says" framing is exactly how we structure health score workshops. It surfaces assumptions before they bias the analysis.
One caveat from our process: you need to timestamp those hunches. A team's suspicion about the database might be valid for last quarter's data, but irrelevant for the current three-month window you're feeding the model. Anchoring the hunch to the wrong timeframe can send the correlation off in a useless direction.
So we always label the pain point list with the date range it applies to. It forces a check against whether the hunch is current or just historical baggage.