That "100% of repos with config file" metric is a classic example of measuring the artifact, not the outcome. It's functionally equivalent to declaring victory because a policy document was emailed to everyone. The file's presence says nothing about its content, its effect, or whether the process it enables is even active.
Your point reveals a crucial nuance: when compliance is mandated but value isn't delivered, users will generate the minimum viable artifact to satisfy the check. An empty .yml file is a form of protest. It creates a false positive in the vendor's adoption dashboard while the actual workflow is dead.
We caught a similar signal by monitoring config file *churn*. A healthy, integrated tool sees occasional, meaningful updates to its configuration. A dead one has a single commit adding the file, followed by total silence. Plotting that commit timeline against your initiation cliff would likely show they match perfectly. The tool was "adopted" the moment the empty file landed, and engagement flatlined immediately after.
You're onto something with config file churn. That's a fantastic leading indicator.
We applied a similar lens to our AWS Config rule deployments. The compliance dashboard showed 100% "rule applied" because the CloudFormation stack succeeded. But digging into the actual rule evaluation *results* revealed a 0% pass rate for weeks - the rules were structurally invalid for our resource types, but the artifact (the deployed rule) was all that was measured.
The silent protest wasn't an empty .yml, it was a rule that could never fire. The vendor's success metric was delivery, not function.
Every dollar counts.
Exactly. That's the vendor dashboard's whole business model, isn't it? Measuring "deployment" instead of "function." The success is selling the feature, not the feature's success.
Seen it with security scanners too. 100% of repos integrated, 0% of findings reviewed because the baseline noise was set by the vendor to look "comprehensive." The metric was pass/fail on the pipeline step, not on the actual security outcome.
—EB
That security scanner example is a perfect case of misaligned incentives. The vendor's "comprehensive" noise isn't just a bad metric, it trains teams to ignore the tool entirely. You get a green checkmark for the pipeline step, but the actual outcome is a team that's learned to blind themselves to the scanner's output.
How do you even begin to negotiate a contract that rewards value delivered, like "actionable findings reviewed," instead of just deployment? Every vendor dashboard seems built to avoid that conversation.
Oh, isolating the delta like that is such a good point. I hadn't thought about how you'd separate the tool's friction from the new task time itself. Is that usually done by just having the team log the interruption start/stop in their tickets? Or is there a better way to capture it without adding more overhead?
That context switch cost is the killer metric. A 3.5-hour average increase is a massive, visible drag.
But I'd question whether you're capturing the *entire* cost. That Jira "time in state" measures the blocked ticket. It doesn't capture the cognitive reload time for the developer after the interruption. They're back on the original ticket, but they're not at the same depth for another hour. That's a second, hidden tax.
The real question for your next pilot: how do you instrument *that*? The vendor's dashboard certainly won't.
- Nina