I've been experimenting with LLMs for generating personalized cold emails, but found most evaluation frameworks too generic. They check for grammar or sentiment, but miss the mark on what makes a *good* sales email. So, I built a more targeted rubric and I'm open-sourcing it here for feedback.
It's a simple JSON schema that scores emails across a few key dimensions I care about:
- **Relevance & Research Depth**: Does it mention a specific, accurate detail about the prospect's company or role?
- **Value Proposition Clarity**: Is the offer or next step immediately clear and tied to a prospect pain point?
- **Tone & Authenticity**: Does it avoid sounding like a generic blast? (This uses a custom classifier prompt).
- **Call-to-Action (CTA) Strength**: Is the CTA specific, low-friction, and time-bound?
Here's the core scoring structure:
```json
{
"email_evaluation": {
"scores": {
"relevance": {"score": 0-5, "evidence": "string"},
"value_proposition": {"score": 0-5, "evidence": "string"},
"tone": {"score": 0-5, "evidence": "string"},
"cta": {"score": 0-5, "evidence": "string"}
},
"overall_score": "float",
"verdict": "PASS/FAIL",
"critical_failures": ["array_of_strings"]
}
}
```
I run this by having an LLM (like GPT-4 or Claude) evaluate each generated email against the rubric, using specific guidelines for each score. The "critical_failures" field catches instant disqualifiers, like getting the company name wrong.
I've been using it to compare different prompts and models, and it's helped me tune my approach significantly. For example, I found that including a one-line instruction like "Reference a recent company blog post" in the prompt boosted the average `relevance` score from 2.1 to 4.3.
Has anyone else built similar domain-specific eval frameworks? I'm particularly curious about:
- How you handle the inherent subjectivity of something like "tone."
- Whether you've integrated similar rubrics into an automated pipeline (like via Coda or Zapier).
- Any ideas for additional dimensions I should consider.
The full rubric with the prompt templates and a small sample dataset is on GitHub. I'd love any critiques or collaboration.
Interesting that your rubric's first metric is "accurate detail about the prospect's company." You're feeding an LLM with presumably scraped or purchased data to generate this personalization, then using another LLM to judge if the first one did it right. How many times have you seen an AI confidently fabricate a "specific, accurate detail" when the source data was thin? You're essentially layering automation on top of shaky ground truth. I'd argue the cost of a wrong detail isn't just a low score, it's a prospect who thinks you're incompetent. Have you built in any validation loop, or is it just two AIs agreeing with each other in a vacuum?
Your k8s cluster is 40% idle.
That's a really neat approach. I've been looking at ways to measure the quality of automated outreach, and most of what I see is super basic, like you said.
Is there a reason you went with a 0-5 scale for each dimension instead of a simpler pass/fail? I'm wondering how you plan to set the threshold for the overall verdict - like, what does the final PASS/FAIL actually look at?
I love seeing frameworks built around actual sales outcomes, not just grammar checks. The "Tone & Authenticity" dimension using a custom classifier is key - that's where most automated systems fall flat and sound robotic.
On the CTA point, I'd add one more sub-criteria: is the CTA something the recipient can do *within their existing workflow*? Asking for a calendar booking is standard, but asking them to log into a new portal or find a specific report adds friction. Might be worth a note in the evidence field.
Do you have sample scores for what constitutes a failing grade vs. a passing one on each dimension? Like, is a 2 in "Relevance" an auto-fail?