Hey everyone, I've been tasked with improving the evaluation and monitoring for our customer-facing chatbot, which is built with LangChain. We're a small team and I'm relatively new to DevOps/MLOps, so I'm looking for some real-world insights.
I've been reading about LangSmith and it looks promising for tracing, debugging, and evaluating LLM calls. But most examples I see are for prototyping. Is anyone here actually using it in a production setting for a live chatbot? I'm especially curious about:
* How you integrated it into your CI/CD pipeline. Do you run evaluations on each deployment? We use GitHub Actions, so any example configs would be awesome.
* What you're actually monitoring in production. Are you tracking latency, token usage, and cost per chain, or more focused on correctness via feedback?
* Any major pitfalls or performance hits you ran into when enabling tracing for all user conversations.
For context, our app is containerized with Docker, and we're planning to deploy to Kubernetes soon. I'm trying to figure out if adding LangSmith is worth the complexity and cost for us, or if we should stick with more basic logging and metrics for now.
Here's a super basic GitHub Actions snippet I was playing with for evaluation—is this even close to how you'd use it?
```yaml
name: Run LangSmith Evaluation
on: [push]
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Run Evaluation Dataset
env:
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
run: |
python scripts/evaluate_chatbot.py
```
Would love to hear about your experiences and any gotchas you've encountered.
Learning by breaking