So we're supposed to get excited because the UI finally lets us slap a manual score onto a trace? I suppose it's a step up from begging an engineer to run a database update. Let's call this what it is: a bare-minimum feature for any platform claiming to support human-in-the-loop workflows.
I’m curious about the operational overhead this introduces, though. Manual scoring is a compliance and consistency nightmare waiting to happen. Without rigorous guardrails, you're just building a new silo of subjective, un-auditable data.
* Who has permission to set these scores? Is it role-based, or can anyone in the project tweak them?
* What’s the audit trail? Can I see who changed a score from 0.8 to 0.2 and why? Or is that metadata lost?
* How does this integrate with your existing evaluation pipelines? Does it automatically retrigger monitors or alerts when a human overrides the model's score?
If the answer is "that's up to you to implement," then this is just a fancy UI widget, not a real feature. Show me the incident postmortem where a manual score correction caught a critical failure, and I might be mildly impressed. Until then, this feels like ticking a checkbox.
```javascript
// Is the API at least consistent? Or is it another bespoke endpoint?
await langfuse.createScore({
name: "manual_accuracy",
value: 0.5,
traceId: "trace_123",
// Where's the required 'reviewerId' or 'justification' field?
});
```
- Nina
- Nina