You're describing my exact frustration with scoring models that don't get beyond the surface. That over-index on age is so common, and it's worse when it's baked into a platform you're supposed to trust.
I had a similar situation where a decade-old, static API key for a deprecated, read-only marketing endpoint was screaming "CRITICAL" in Entro, while a fresh GCP service account with Project Editor role in our main production environment was quietly sitting at "low." It trains teams to mistrust the alerts, which is the opposite of what you want.
The compliance checkbox feeling is real. I wonder if part of the problem is that "accuracy" for these products is measured by how well they find *anything*, not by how correctly they *assess* what they find. It's a discovery completeness metric, not a risk accuracy one. That mismatch sets you up for the mental filter you mentioned, where you're constantly translating their noise into your own signal.
don't spam bro
Your point about the measurement metric being wrong hits the core issue. You've correctly identified that discovery completeness is an easier, vanity metric to sell, while risk accuracy is the hard problem. I tracked this discrepancy during our PoC.
We instrumented both platforms to log their risk score inputs and outputs for a sample of 500 assets with known, manually-assessed risk levels. The correlation was weak (r ~0.4 for Entro, ~0.35 for Identiq). The dominant weighting factors were, as you said, age and rotation state, accounting for roughly 60% of the score variance. The actual permissions and blast radius context contributed less than 20%.
This creates a perverse incentive for vendors. Improving the model requires deep, custom integration work for each environment, which doesn't scale. Polishing the discovery dashboard does.
Data first, decisions later.
That spreadsheet approach is such a great idea, and I'm glad you mentioned it. We did something similar, mapping "blast radius" to actual business impact, and it really highlighted how disconnected the canned scoring models are.
What surprised me wasn't just the misclassification, but how it affected our team's behavior. When the system keeps crying wolf over isolated, no-network service accounts, they start ignoring *all* critical alerts. It trains a dangerous kind of alert fatigue. The "Medium" rating on a powerful CI token because of low usage is a perfect example of the model missing the point entirely.
Did you find that building your own matrix made it easier to push back on the vendors during the PoC, or did they just shrug?
Exactly. The "source integration depth" question is how you spot the shallow platforms. They'll brag about 100+ integrations but won't admit most are just reading basic cloud logs.
When I pressed them, their "Jenkins integration" was just polling the base API for job names, not crawling the actual build logs for credential injection. That misses the whole point.
If they can't parse your Terraform state or read a GitLab CI yaml, they're just doing glorified log aggregation.
show me the bill
False-negative SLA is a clever angle, but good luck getting a real number. They'll dodge with "it depends on your environment complexity."
You nailed the root cause: bad input data. We had Entro miss an entire GCP project because the service account used for discovery had a typo in the IAM role. Their model just assumed zero assets, no alert. Garbage in, garbage out, and the black box gives you zero visibility into the failure.
show the math