In my ongoing evaluation of CRM content automation suites, a common vendor claim is "enterprise-grade AI content generation." To test this objectively, I needed a scalable, repeatable method to score AI output quality, moving beyond subjective "this looks good" assessments. The project involved generating 1,000 product descriptions across three platforms (using their native AI tools) and scoring them against a fixed rubric. Manual evaluation was impossible at that scale, so I built an automated scoring pipeline.
The core of the methodology was a multi-dimensional evaluation framework, operationalized through a combination of rule-based checks and LLM-as-a-judge prompts. The rubric consisted of five weighted dimensions:
* **Instruction Adherence (30%):** Did the output follow the specific commands in the prompt (e.g., include key features, use a bulleted list, adhere to a specified word count)? This was partially automated with simple string matching and regex for structural elements.
* **Factual Accuracy (25%):** For product descriptions, this meant correctly including product specifications (SKU, dimensions, material) provided in the source data. I used a dedicated NER (Named Entity Recognition) model to extract claimed specs and compared them to the source CSV.
* **Brand Voice Consistency (20%):** This required a reference benchmark. I created a vector database of 50 human-written, approved descriptions. The cosine similarity between the embedding of the AI output and its nearest neighbor in the reference set provided a quantitative score.
* **Readability & Grammar (15%):** Automated via standard libraries (like `textstat` for Flesch-Kincaid, and language tool checkers for basic grammar).
* **Creativity & Uniqueness (10%):** Measured by calculating the perplexity of the text using a mid-sized language model; higher, controlled perplexity could indicate less generic phrasing. Also included a self-similarity check across the entire 1,000-description set to penalize repetitive templates.
The most nuanced dimension, "Overall Coherence and Persuasive Quality," required an LLM-as-a-judge. I configured a GPT-4 judge with a detailed system prompt outlining the scoring rubric (1-5 scale) and provided examples of high and low scores for each dimension. The key was structuring the judgment call as a structured JSON output. Each description was sent to the judge API with the original product specs and brand voice guidelines appended to the prompt.
The pipeline was orchestrated in Python. The process flow was:
1. Input: Raw product data (CSV) and generation prompts sent to three AI content APIs.
2. Generation: Collected all 3,000 outputs (1,000 per platform) into a master dataframe.
3. Automated Scoring: Ran each output through the rule-based modules (Adherence, Accuracy, Readability, Uniqueness), storing scores.
4. LLM Judgment: Batched outputs and called the judge API for the coherence/persuasion score and a qualitative rationale.
5. Aggregation: Compiled all dimensional scores into a weighted total for each description, then averaged by platform.
The results were revealing. Platform A excelled in factual accuracy (98.7%) but scored lowest in creativity. Platform B had near-perfect instruction adherence but showed significant brand voice drift in 22% of outputs. Platform C, while generally balanced, had a higher rate of repetitive structures. This data-driven approach provided a clear, comparative benchmark far more actionable than any vendor-provided case study. The framework itself, while built for product descriptions, is now being adapted to evaluate AI-generated email sequences and knowledge base articles, with adjustments to the rubric weights and reference data.