Hey everyone 👋
Our team has been using Notion AI heavily for drafting internal documentation, runbook summaries, and even first passes at changelog entries. Early on, we realized we needed a consistent way to judge the *quality* of the AI's output beyond just "looks good." We were all giving vague, subjective feedback. To solve this, we built a simple scoring rubric that we now use for any AI-generated content before it gets published or shared.
It's been a game-changer for giving structured feedback and for actually *improving* our prompts over time. I thought I'd share it here in case it helps anyone else.
We score from 1 (Poor) to 5 (Excellent) across five categories. The key is that we have specific, observable criteria for each score.
### Our Rubric (SQuACC)
**1. Substance & Accuracy (S)**
* **5:** Factually flawless for our context. Adds non-obvious, relevant detail.
* **3:** Correct on basics, but shallow or misses nuanced/internal knowledge.
* **1:** Contains clear inaccuracies or "hallucinations" about our systems/processes.
**2. Quality of Language (Q)**
* **5:** Reads like a polished human wrote it. Flows naturally, with varied sentence structure.
* **3:** Grammatically correct but robotic. May overuse certain phrases or be overly simplistic.
* **1:** Awkward phrasing, repetition, or grammar errors that need a full rewrite.
**3. Actionability & Structure (A)**
* **5:** Logically structured (e.g., problem -> steps -> verification). Clear calls to action or decision points.
* **3:** Information is present but poorly organized. Lacks clear hierarchy or sequencing.
* **1:** Disorganized, illogical flow. Impossible to follow for a task.
**4. Context Adherence (C)**
* **5:** Perfectly tailors tone & detail to the specified audience (e.g., new hire vs. architect).
* **3:** Generic, one-size-fits-all tone. Doesn't leverage the provided source material effectively.
* **1:** Ignores explicit instructions about audience, style, or input context.
**5. Completeness (C)**
* **5:** Addresses all explicit and implicit requirements. Needs no further expansion.
* **3:** Covers the main points but omits minor, expected elements or examples.
* **1:** Clearly incomplete or cuts off mid-thought.
### How We Use It
After generating a block of text, the reviewer scores each category and leaves brief comments. We often calculate a quick average, but the individual scores are more useful. Here's a mini example from a recent incident post-mortem draft:
```markdown
**Prompt:** "Using the provided outage timeline, draft a 'Root Cause Analysis' section for engineers in a blameless tone."
**SQuACC Scores:**
- **S:** 4 (Timeline facts correct, but root cause oversimplified)
- **Q:** 5
- **A:** 3 (Jumps between timeline and cause without clear sub-sections)
- **C:** 4 (Tone is good, but too technical for intended wider audience)
- **C:** 5
**Avg:** 4.2
**Feedback to Iterate:** "Great start. Please restructure as: 1. Immediate Trigger, 2. Underlying System Flaw, 3. Contributing Factors. Soften the explanation of the database timeout logic for a general engineering audience."
```
This structured approach lets us move from "make it better" to "improve Actionability from a 3 to a 4 by adding sub-headers." It also highlights which areas Notion AI consistently underperforms in for our use cases (for us, it's often **Context Adherence** unless the prompt is extremely detailed).
Does anyone else use a similar scoring system? I'd be curious if you have different categories or criteria. This has definitely made our AI-aided writing more of a science and less of an art.
— francesc
— francesc