We're embarking on a significant LLM integration project, aiming to move several customer-facing support and internal knowledge retrieval systems from rule-based to generative AI. The stack is primarily on AWS (Bedrock, some SageMaker), with deployment targets being Lambda and ECS. As the team responsible for the operationalization and reliability of these systems, we need to establish a robust, repeatable evaluation framework from day one.
Our requirements are specific: we need something that integrates cleanly into our existing CI/CD pipelines (GitHub Actions), can handle both automated/metrics-based evaluation (using another LLM as judge) and human-in-the-loop review, and stores results in a queryable format (preferably S3/Athena or a simple RDS). Cost is a factor, but developer time and maintainability are higher priorities. We've looked at the usual suspects—LangSmith, Phoenix, RAGAS, and the more recent OpenAI Evals—but each seems to have a different philosophical leaning and operational overhead.
I'm leaning towards an open-source core for maximum control and to avoid vendor lock-in at the evaluation layer. My current thinking is a two-tiered approach:
1. **Lightweight, automated regression testing** for every pull request using a framework like `ragas` or `phoenix`.
2. **Periodic, comprehensive evaluation suites** run as part of a scheduled pipeline, perhaps leveraging a more customizable framework.
Here's a strawman for the automated tier using a hypothetical test:
```python
# Example of what we'd want to codify for a retrieval pipeline
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall
# After running our pipeline which outputs `answer`, `contexts`, `question`
dataset = ...
results = evaluate(
dataset,
metrics=[faithfulness, answer_relevancy, context_recall],
llm=bedrock_llm, # Configured for Claude Haiku
embeddings=bedrock_embeddings # Using Titan
)
generate_report(results, output="s3://our-bucket/eval-runs/pr-123/")
```
Key architectural questions for the community:
* Is there a framework that excels at managing and versioning **evaluation datasets** themselves (e.g., golden sets of Q&A, edge cases)?
* How do you handle the **cost and latency** of running evaluations that themselves use LLMs (like GPT-4 or Claude Sonnet as a judge)? Is there a standard pattern for caching or using smaller, cheaper models for a first pass?
* For those on AWS, have you successfully integrated any of these open-source frameworks with Bedrock's models natively, avoiding unnecessary proxies or adapters?
I'm wary of building a bespoke monster, but also of adopting a framework that becomes a black box. The ideal would be something declarative, where we can define evaluation scenarios and scoring rubrics as code, track drift over time, and tie results directly to our observability stack (CloudWatch, Prometheus).
--from the trenches
infrastructure is code