I've been using PromptLayer to track LLM call performance and costs across several microservices, which has been generally positive. However, the implementation of the 'scores' feature feels like a missed opportunity for deeper analysis.
The ability to log a numeric score (e.g., `promptlayer.scoring.log(score=0.85)`) is useful for high-level tracking. But in a production backend, a score is meaningless without the context that produced it. For a database latency query, you'd never just log `120ms`; you'd attach the query fingerprint, parameters, and host. Similarly, I need to attach custom metadata to these LLM scores.
**My concrete use-case:** I'm scoring the quality of a generated SQL query from an LLM. The score alone tells me little. I need to attach:
* The raw user question
* The final, executed SQL (for validation)
* Which `pg_stat_statements` query hash it maps to
* The database execution time (to correlate LLM score with actual DB performance)
Currently, I'm forced to log this metadata separately and try to link it via `request_id`, which is fragile and breaks any built-in aggregation PromptLayer might offer.
```python
# What I can do:
promptlayer.scoring.log(score=sql_quality_score)
# What I need:
promptlayer.scoring.log(
score=sql_quality_score,
metadata={
"user_query": user_input,
"generated_sql": final_sql,
"pg_query_hash": query_hash,
"db_execution_ms": db_latency
}
)
```
Without this, the scores are just isolated numbers. I can't slice the data post-hoc to answer questions like "Do higher-scoring queries also have lower database latency?" or "Which user questions consistently produce low-scoring queries?"
Has anyone built a workaround for this? Perhaps a pattern using tags or the `pl_tags` field in a creative way? Or are we all just maintaining a separate analytics pipeline for this correlation?
-- latency
sub-100ms or bust
You're right, context is everything for a score. It's an unactionable metric without it. In incident management, we'd call that a dead-end signal.
Your SQL example is spot on. I've seen teams handle this by wrapping the scoring call. They create a local function that logs the full metadata to their own system, then calls the vendor's logging with just the score. You still have the link, but you own the context.
The `request_id` linking you mentioned is fragile because it relies on two systems agreeing on an ID. If PromptLayer doesn't expose that ID in their UI for the score, the link is broken on their side.
null