Skip to content
Notifications
Clear all

Sharing: My team's rubric for scoring AI-generated content quality

13 Posts
13 Users
0 Reactions
75 Views
(@francesc)
Reputable Member
Joined: 4 months ago
Posts: 286
Topic starter   [#23927]

Hey everyone 👋

Our team has been using Notion AI heavily for drafting internal documentation, runbook summaries, and even first passes at changelog entries. Early on, we realized we needed a consistent way to judge the *quality* of the AI's output beyond just "looks good." We were all giving vague, subjective feedback. To solve this, we built a simple scoring rubric that we now use for any AI-generated content before it gets published or shared.

It's been a game-changer for giving structured feedback and for actually *improving* our prompts over time. I thought I'd share it here in case it helps anyone else.

We score from 1 (Poor) to 5 (Excellent) across five categories. The key is that we have specific, observable criteria for each score.

### Our Rubric (SQuACC)

**1. Substance & Accuracy (S)**
* **5:** Factually flawless for our context. Adds non-obvious, relevant detail.
* **3:** Correct on basics, but shallow or misses nuanced/internal knowledge.
* **1:** Contains clear inaccuracies or "hallucinations" about our systems/processes.

**2. Quality of Language (Q)**
* **5:** Reads like a polished human wrote it. Flows naturally, with varied sentence structure.
* **3:** Grammatically correct but robotic. May overuse certain phrases or be overly simplistic.
* **1:** Awkward phrasing, repetition, or grammar errors that need a full rewrite.

**3. Actionability & Structure (A)**
* **5:** Logically structured (e.g., problem -> steps -> verification). Clear calls to action or decision points.
* **3:** Information is present but poorly organized. Lacks clear hierarchy or sequencing.
* **1:** Disorganized, illogical flow. Impossible to follow for a task.

**4. Context Adherence (C)**
* **5:** Perfectly tailors tone & detail to the specified audience (e.g., new hire vs. architect).
* **3:** Generic, one-size-fits-all tone. Doesn't leverage the provided source material effectively.
* **1:** Ignores explicit instructions about audience, style, or input context.

**5. Completeness (C)**
* **5:** Addresses all explicit and implicit requirements. Needs no further expansion.
* **3:** Covers the main points but omits minor, expected elements or examples.
* **1:** Clearly incomplete or cuts off mid-thought.

### How We Use It

After generating a block of text, the reviewer scores each category and leaves brief comments. We often calculate a quick average, but the individual scores are more useful. Here's a mini example from a recent incident post-mortem draft:

```markdown
**Prompt:** "Using the provided outage timeline, draft a 'Root Cause Analysis' section for engineers in a blameless tone."

**SQuACC Scores:**
- **S:** 4 (Timeline facts correct, but root cause oversimplified)
- **Q:** 5
- **A:** 3 (Jumps between timeline and cause without clear sub-sections)
- **C:** 4 (Tone is good, but too technical for intended wider audience)
- **C:** 5
**Avg:** 4.2

**Feedback to Iterate:** "Great start. Please restructure as: 1. Immediate Trigger, 2. Underlying System Flaw, 3. Contributing Factors. Soften the explanation of the database timeout logic for a general engineering audience."
```

This structured approach lets us move from "make it better" to "improve Actionability from a 3 to a 4 by adding sub-headers." It also highlights which areas Notion AI consistently underperforms in for our use cases (for us, it's often **Context Adherence** unless the prompt is extremely detailed).

Does anyone else use a similar scoring system? I'd be curious if you have different categories or criteria. This has definitely made our AI-aided writing more of a science and less of an art.

— francesc


— francesc


   
Quote
(@alice2)
Estimable Member
Joined: 4 months ago
Posts: 182
 

This is a fantastic framework, and I'm glad you shared it. The move from subjective "looks good" to observable criteria is exactly what turns casual use into a measurable process. I'm particularly interested in how you're using the scores to *improve your prompts over time*.

Have you formalized that feedback loop? For instance, are you tagging low-scoring outputs with the specific failing criteria (like "Substance & Accuracy: 2") and then systematically revising the initial prompt to address that gap? Without that structured retrospection, there's a risk the rubric just becomes a grading tool rather than a developmental one.

Also, how does your team handle the inevitable score variance between reviewers on categories like "Quality of Language"? We've found establishing a few concrete style guide anchors - sentence length, passive voice avoidance, forbidden jargon - necessary to get consistent calibration.


Your data is only as good as your pipeline.


   
ReplyQuote
(@carlosp)
Reputable Member
Joined: 4 months ago
Posts: 255
 

You've raised two critical operational points. On the feedback loop, we absolutely formalized it. We log every scored output in a simple Airtable base with columns for the prompt text, the SQuACC scores, and the specific failing criteria noted as tags. We run a weekly review where we cluster prompts by low-score patterns, which directly informs our prompt library revisions. For example, a cluster of "Substance: 2" scores on runbook drafts led us to add a mandatory "include specific error codes from our monitoring system" clause to that prompt template.

Regarding score variance on "Quality of Language," we found that without calibration, it was too subjective. We solved it by creating a shared benchmark set of five sample outputs that we all scored independently, then discussed discrepancies to establish a common internal standard. Now, for any borderline score, the reviewer must cite which benchmark sample the output most closely resembles. It's not perfect, but it reduced variance significantly. Have you considered a similar calibration exercise?


show me the SLA


   
ReplyQuote
(@infra_architect_42)
Honorable Member
Joined: 5 months ago
Posts: 367
 

Your approach to formalizing the feedback loop with Airtable and weekly pattern analysis is the most valuable part of this. It mirrors how we treat infrastructure configuration drift, where logging anomalies without a scheduled review process is just noise.

The benchmark calibration for language quality is smart, but I'd add a caveat from similar exercises in network policy reviews. A static benchmark set can become outdated as the AI model or your internal style guide evolves. We schedule a quarterly "benchmark refresh" where we collectively re-score the old samples and introduce new ones, which keeps the calibration meaningful.

Have you considered applying statistical measures, like calculating inter-rater reliability scores during your calibration discussions, to quantify how well your team's scoring is actually converging?


Boring is beautiful


   
ReplyQuote
(@calebs)
Reputable Member
Joined: 4 months ago
Posts: 318
 

The framework's solid but you're missing a critical category for infrastructure work: actionability. "Substance & Accuracy" covers facts, but a flawless, polished doc can still be useless for an on-call engineer at 3 AM.

I'd add a sixth dimension: "Clarity of Commands & Procedures." A 5 requires specific, executable commands, exact config file paths, and clear success/failure criteria. Otherwise you're just grading a well-written fiction.



   
ReplyQuote
(@data_pipeline_guy_42)
Reputable Member
Joined: 5 months ago
Posts: 271
 

The SQuACC framework is a smart start, especially for general internal docs. But for anything operational - runbooks, deployment playbooks, incident post-mortems - it's incomplete.

You need a separate "Actionability" score. A runbook can score a 5 on Substance and Quality but still be useless if it says "restart the service" instead of giving the exact systemctl command, the service name as it exists in your environment, and what a successful restart log line looks like.

We learned this the hard way. An AI drafted a "Database Connection Pool Drain" procedure that was grammatically perfect and factually correct about the symptoms. It failed because it didn't specify *which* metrics dashboard to check first, leaving the engineer to search. That's a substance accuracy score of 5, but an operational value of 1.

Bake operational specifics into your prompts for that content type. Mandate placeholders for exact command syntax, config file paths, and metric names. Then score against that.


garbage in, garbage out


   
ReplyQuote
(@gracem)
Reputable Member
Joined: 4 months ago
Posts: 294
 

This is such a crucial addition for operational docs. You're absolutely right that "accurate but vague" is a different kind of failure.

> Mandate placeholders for exact command syntax, config file paths, and metric names.

We adopted a similar rule after a few drafts lacked concrete API endpoint examples. Now, our prompt template for runbooks includes a required "### SPECIFIC ARTIFACTS" section where we list the exact tools, commands, and dashboards relevant to that service. The AI has to incorporate them.

One thing we've noticed, though, is that this can sometimes push the AI into being *too* specific and brittle, like hard-coding a server hostname that changes between environments. So we pair those mandatory placeholders with a "### VARIABLES" section for things that need to be filled in manually. It helps keep the draft actionable but also portable.


Automate everything.


   
ReplyQuote
(@ellaj8)
Reputable Member
Joined: 4 months ago
Posts: 295
 

The calibration exercise is the difference between a process that's theater and one that actually works. We did something similar, but we had to add one rule: the benchmark samples must include at least one 'bad' example. If you only calibrate on good outputs, you'll still have wild variance on what constitutes a 2 versus a 3.

Your Airtable logging is smart. We found we had to add a 'prompt lineage' field to track which revised prompt template version generated the output, or else you can't tell if your fixes are actually fixing anything.


Trust but verify – and audit


   
ReplyQuote
(@hiker42)
Reputable Member
Joined: 4 months ago
Posts: 232
 

The prompt lineage field is non-negotiable. It's the only way to measure ROI on the time you spend tweaking templates. Without it, you're just guessing.

Your rule about including a 'bad' benchmark is spot on. We had the same realization. Calibration sessions where everyone scores a '5' are worthless. You need the low scores to force alignment on what failure actually looks like in your context.

We took it a step further: we now have one 'failing' sample for each primary category of our rubric. It forces the team to distinguish between, say, a factual inaccuracy versus a clarity failure.



   
ReplyQuote
(@cost_observer_42)
Honorable Member
Joined: 5 months ago
Posts: 407
 

I appreciate the rigor, but I have to ask: how do you cost the time spent creating and maintaining this 'benchmark fail sample' library? That sounds like a non-trivial ongoing calibration tax.

You've built a system to measure failure types, but have you measured the failure rate of the system itself? I'd be curious if you've ever calculated the marginal cost of a scoring session versus the savings from a marginally improved prompt. Feels like you could be optimizing for rubric purity over actual cost per useful output.


cost_observer_42


   
ReplyQuote
(@cloud_cost_analyst_pro)
Honorable Member
Joined: 8 months ago
Posts: 469
 

Agree on the critical need for actionability. Your database example is perfect. It's the same problem with cloud cost reports. An AI can generate a flawless, accurate report listing "high spend on EC2," but if it doesn't specify the exact instance IDs, region, and the 3-click path to the RI purchase screen in our console, it's useless for the engineer who has to act.

One caveat: adding too many mandated specifics can bloat prompt tokens, which increases direct cost per generation. You have to weigh that against the operational waste of a vague output. We solved it by templating the variables.

Prompt becomes: "For service [SERVICE_NAME], use the exact restart command: `sudo systemctl restart [SERVICE_NAME].service`. Verify via `journalctl -u [SERVICE_NAME].service -n 5`. Success log line contains '[LOG_PATTERN]'." The AI fills the placeholders, and we score missing any of those three concrete artifacts as an automatic fail.


cost per transaction is the only metric


   
ReplyQuote
(@crm_hopper_2025_new)
Honorable Member
Joined: 5 months ago
Posts: 365
 

You're focusing on the scoring, but the bigger issue is you're using Notion AI for this in the first place. It's a vendor-locked ecosystem. What happens when you want to port those 'improved prompts' or your benchmark library to a different platform next quarter? You can't.

Scoring a 5 on 'Quality of Language' inside a walled garden is meaningless if the output is stuck there.



   
ReplyQuote
(@chloem)
Reputable Member
Joined: 4 months ago
Posts: 231
 

That's a really practical question. We faced the same overhead issue and found the cost wasn't in creating the fail samples, but in the recurring calibration meetings. We automated most of it by building the benchmark samples directly into our scoring dashboard.

Now, when you log a score, you see a random 'anchor' example from our benchmark library for comparison. It takes the subjective debate out of the session. The setup time was maybe 10 hours, but it cut our scoring time per doc by about two-thirds.

You're right to question the marginal cost, though. We only run this formal scoring for high-stakes content, like customer-facing runbooks or compliance docs. For an internal blog draft? We'd never bother. The system's value is in preventing one critical operational failure, not in polishing every single output.



   
ReplyQuote