The new feedback API endpoints are a significant step forward. Prior to this, capturing user feedback and linking it back to specific runs was a manual, error-prone process. Now we can actually close the loop between production performance and model tuning.
The key endpoints are straightforward. You can create a feedback object tied to a run ID, and query it later for analysis.
```python
# Example: attaching a binary correction feedback
from langsmith import Client
client = Client()
run_id = "..." # The LLM run you want to score
feedback = client.create_feedback(
run_id=run_id,
key="correction",
score=0.0, # 1.0 for correct, 0.0 for incorrect
comment="The model hallucinated the date."
)
```
What I'm watching:
* **Performance overhead:** How does this scale when ingesting high-volume feedback from live endpoints?
* **Aggregation efficiency:** Can I easily slice feedback scores by metadata (deployment version, prompt hash) to compute reliability metrics?
* **Integration pipeline:** Need to see how this feeds into automated retraining or prompt versioning workflows.
If they've built this with bulk ingestion and efficient time-series queries in mind, it will be genuinely useful. Otherwise, it's just a structured log.
—gp
Data over opinions
The performance overhead question is critical, and it often comes down to their underlying data store choice. Your aggregation point about slicing by metadata makes me suspect they'll need efficient indexing on columns like deployment_version and prompt_hash, not just run_id and timestamp.
If they're using something like Postgres, they could handle it with partial indexes on frequently queried metadata combinations. But for true high-volume time-series feedback, they might be better off with a dual-write strategy - feedback to a transactional store for consistency, then aggregated metrics to a columnar OLAP system for slicing. That's how we've structured similar telemetry pipelines at scale.
Have you checked whether the API supports batch creation of feedback objects? That's usually the first indicator they've considered ingestion load. Without a /feedback/batch endpoint, you'll be limited by HTTP overhead.
SQL is not dead.