What is the best way to measure 'tone' consistency across tools?

Hey everyone 👋

So I’ve been tinkering with a handful of AI writing assistants lately — think Claude, ChatGPT, Gemini, and some of the newer open-source models via various web interfaces and API wrappers. I’m trying to settle on one for a long-form technical documentation project, and one of my biggest headaches isn’t factual accuracy or structure… it’s **tone consistency**.

We all know each tool has its own “voice,” but when you’re feeding it a series of related prompts or a long document to refine, how do you actually *measure* whether the output stays in the same stylistic lane? I’m not just talking about formal vs. casual — I mean subtle stuff like:

* Sentence length variation (or lack thereof)
* Preference for active vs. passive voice
* Density of transitional phrases
* Use of certain adjectives or hedging language (“might,” “could,” “often”)
* How it handles lists, code block introductions, or warnings

I’ve tried the naive approach: just reading the outputs side-by-side and feeling it out. That’s… not scalable or objective. I’ve also fed outputs back into the tools themselves with prompts like “Analyze the tone of this text,” but then you’re just measuring one AI’s *opinion* of another AI’s tone, which feels circular.

My current experiment involves using some programmatic linting for prose. For example, I’ll run outputs through a Python script that uses libraries like `textstat` for readability scores, or `spaCy` for part-of-speech tagging to get crude metrics.

```python
import textstat
import spacy

# Load a sample output from Tool A and Tool B
sample_a = "This function initiates the process. It is recommended to verify the settings first."
sample_b = "Fire up this function to start the process. Don't forget to check your settings!"

# Flesch reading ease comparison
print(f"Tool A Flesch: {textstat.flesch_reading_ease(sample_a)}")
print(f"Tool B Flesch: {textstat.flesch_reading_ease(sample_b)}")

# Load NLP model for deeper analysis
nlp = spacy.load("en_core_web_sm")
doc_a = nlp(sample_a)
doc_b = nlp(sample_b)

# Compare sentence lengths
print(f"Avg sentence length A: {sum(len(sent) for sent in doc_a.sents)/len(list(doc_a.sents))}")
print(f"Avg sentence length B: {sum(len(sent) for sent in doc_b.sents)/len(list(doc_b.sents))}")
```

But this feels incomplete. It catches extremes, but misses the nuanced “vibe.”

So I’m curious: has anyone here set up a more robust framework for this? Perhaps using:

* **Embedding vectors** from sentence-transformers to measure semantic *and* stylistic drift across paragraphs?
* **Custom regex patterns** to flag unwanted phrasing tics?
* **Fine-tuned classifiers** trained on your own “good tone” vs. “bad tone” examples?
* A **checklist of stylistic rules** you enforce manually (which defeats the automation purpose, but maybe it’s necessary)?

What’s your testing rig look like? Are you just accepting that you’ll need a human editor to smooth things over in the final pass, or have you found a way to get truly consistent, tool-agnostic tone by prompt engineering alone? Share your setups and war stories!