So we're supposed to trust the numbers you're reporting on yourselves. How many appeals were denied versus overturned? What's the actual turnaround time on a ticket, not the average you cherry-pick? Show the raw escalation paths.
I see a lot of "policy updates" but nothing on moderator consistency. One person's heated debate is another's harassment, depending on who's on shift. Where's the data on that?
Just saying.
You're right to ask for the distribution, not just the average. A single mean turnaround time is almost useless without the variance and the outliers. If the 95th percentile is weeks, that tells a very different story than if it's a few days, even with the same average.
The consistency question is even more critical from a data modeling perspective. It's a classic inter-rater reliability problem. The report mentions policy updates, but without measuring agreement between moderators on similar cases - perhaps through anonymized test cases or a review of historical decisions - there's no way to quantify consistency. Policy updates don't address the underlying measurement gap.
They should be publishing metrics like Cohen's kappa on a sample of moderated content. Anything less is just reporting outputs without auditing the quality of the decision-making process itself.
Data doesn't lie, but folks sometimes do.
Totally agree about distribution. I've seen dashboard averages hide all sorts of problems - a smooth 24-hour SLA can mask a pile of 2-week-old tickets no one wants to touch.
Love the inter-rater reliability point. Cohen's kappa or even a simple percentage agreement on a sample of flagged content would be so revealing. Policy updates feel good, but if the kappa is low, you haven't solved the core issue, you've just moved the goalposts. It's like changing the questions on a test because graders can't agree on the answers.
data over opinions
Spot on about the kappa. That metric would cut through the noise.
But in practice, teams rarely collect the data needed to calculate it. You'd need a system that logs which moderator saw each case *first* and then logs every review from an escalation. Most ticket systems just overwrite the assignee. You can't measure disagreement if you don't capture the disagreement trail.
So the problem isn't just refusing to publish the stat. It's often that their integration is too shallow to even generate it. They'd need to hook their mod tools into a proper audit log, which is more middleware work than they usually budget for.
Integration is not a project, it's a lifestyle.