Showcase: We built a custom scorer for our internal policy checks

Another day, another vendor blog post about "revolutionizing" LLM evaluation with their shiny platform. But let's cut through the hype. The real value isn't in their pre-baked metrics—it's in being able to build what *you* actually need.

We've been using Freeplay for a few months, mostly for prompt iteration. The standard scorers (factual, PII) are fine for demos, but we have a ton of internal compliance and style rules they don't cover. Instead of manually checking every conversation sample, we built a custom scorer to automate our policy checks. Surprisingly, the custom scorer API is the most sensible part of their system.

Here's the gist. We needed to flag responses that referenced specific internal project codenames (which are confidential) and check for adherence to our required disclaimer phrasing. We built a simple Python function that uses regex patterns and keyword checks. The key was plugging it into their evaluation runs.

```python
import re

def internal_policy_scorer(response_text: str, **kwargs) -> dict:
"""
Returns a score of 1.0 if it passes all internal checks, 0.0 otherwise.
Also returns a list of violated rules.
"""
violations = []

# Check for confidential project codenames
confidential_terms = ["PROJECT-ARGONAUT", "PROJECT-SILVERBACK"]
for term in confidential_terms:
if term.lower() in response_text.lower():
violations.append(f"Mentions confidential term: {term}")

# Mandatory disclaimer check
required_disclaimer = "For official use only."
if required_disclaimer not in response_text:
violations.append("Missing required disclaimer.")

# Score is 1.0 if no violations, else 0.0
score = 1.0 if len(violations) == 0 else 0.0

return {
"score": score,
"metadata": {
"violations": violations,
"passed": score == 1.0
}
}
```

You wrap this in their decorator, push it to your Freeplay instance, and suddenly you can run batch evaluations across hundreds of test cases and instantly see which responses violate internal policy. The integration was straightforward, which I'll admit was a relief.

The catch? Now we're locked into their runtime for these evaluations. If we want to run this scorer outside Freeplay, we have to strip out their decorator and maintain a separate version. Typical vendor lock-in, even for the "custom" stuff. It works, but it's another piece of architecture you can't easily take with you.

Just my 2 cents