I’ve been looking at PromptLayer for logging our OpenAI calls, and I keep seeing the “score a response” feature mentioned. I get the basic idea—you can attach a score to an API response—but I’m fuzzy on the specifics.
What exactly am I measuring when I use this? Is it just a simple 1-10 for quality, or can it be something custom like “accuracy on a scale of 1-5”? And where does that score data go—can I graph it over time or tie it to a specific prompt version? A real-world example would really help.
Exactly what you measure is totally up to you, that's the power of it. It can be a simple 1-10, a 1-5 for accuracy, or even a custom tag like "helpful/unhelpful." You define the rubric based on what matters for your use case.
The data gets stored with that specific API call log in your PromptLayer dashboard. So later, you can filter and view all responses with a score below 3, or graph the average score for a particular prompt template over the last month. It's incredibly useful for spotting drift or testing prompt variations.
A real example: we score our customer support auto-replies on "correctness" and "tone." If the score drops, we know to tweak the prompt.
That's a solid practical example. It's key to tie those scores to actual business metrics, like you did with customer support.
The danger is creating a vague "quality" score. If the rubric isn't operationally defined, like what a "correctness" of 4 versus 5 actually means, the data becomes noise. You can't act on drift you can't define.
Your point about filtering logs for scores below a threshold is where this pays off. That's your direct pipeline to failed cases for root cause analysis.
Less spend, more headroom.
Exactly. The operational definition is everything. It's the difference between a metric you can alert on and one that's just a dashboard decoration.
In FinOps we run into this all the time. A vague "cost efficiency" score is useless. But a score defined as `(1 - (unused_reservation_costs / total_reservation_costs)) * 10` is actionable. If that drops below 8, we have a specific alert to go check our coverage.
Your "correctness of 4 vs 5" point is spot on. That definition needs to live in a runbook, not just someone's head. Otherwise, you can't automate the next step. Are you paging an engineer at 3am for a score of 3, or just flagging it for review? The rubric tells you.