Our security team, in a fit of what I can only describe as "benchmark-driven euphoria," decided to deploy risk-based conditional access policies across our entire Entra ID tenant last quarter. The pitch was classic vendor metrics: "95% detection accuracy, 99.9% assurance!" After three months of operational chaos and a revolt from the finance department—who were repeatedly locked out during month-end closing—we've now disabled the feature entirely. I suspect our experience is not unique, and it highlights a fundamental methodological flaw in how these risk signals are evaluated.
The core issue is the same one plaguing many AI-driven security products: the training data and the evaluation benchmarks are catastrophically misaligned with real-world, heterogeneous enterprise environments. The "risk" score is a black-box model, likely trained on a dataset that doesn't reflect our specific user behavior patterns, application usage, and network topology. The false positive rate in production was orders of magnitude higher than any reasonable baseline.
Our primary pain points included:
* **Location-based false positives:** Employees using reputable, corporate-approved VPNs for privacy were flagged as "impossible travel" or "anonymous IP address."
* **Application-specific overreach:** Legacy internal applications with non-standard authentication patterns triggered "unfamiliar sign-in properties" constantly for the same users, day after day.
* **The cost of remediation:** The overhead shifted from the security team (investigating true positives) to the help desk (resetting MFA, unlocking accounts) and ultimately to the end-users (lost productivity). The total cost of ownership calculation was never run with realistic false-positive rates.
Our conditional access policy configuration looked deceptively simple, which is part of the problem. It gave a false sense of control.
```json
{
"displayName": "High-Risk Sign-In Block",
"state": "enabled",
"conditions": {
"userRiskLevels": ["high"],
"signInRiskLevels": ["high", "medium"],
"applications": {"includeApplications": ["All"]}
},
"grantControls": {"operator": "AND", "builtInControls": ["block"]}
}
```
The simplistic binary logic—"if risk = high, then block"—fails to account for necessary nuance. There was no graceful degradation path, no ability to say "require MFA for medium risk from a trusted device, but block high risk only from new devices."
My contention is that risk-based policies, as currently implemented, are akin to running a benchmark suite on a clean lab machine and then applying the scoring to a messy, production multi-tenant environment. The results are meaningless and disruptive. Has anyone else conducted a similar "experiment" and gathered quantitative data on lockout rates? I'm particularly interested in the differential impact across departments (e.g., engineering vs. sales vs. finance). Did you find a way to tune the sensitivity effectively, or did you also revert to deterministic rules based on device compliance and trusted locations?