Hey everyone,
I’ve been following the discussions here on eval frameworks with a lot of interest. I’m at a point where I want to start systematically evaluating some LLM outputs for a side project—mostly comparing different prompts and models for a customer support chatbot scenario.
Looking at tools like LangChain’s evaluators, DeepEval, and even the OpenAI Evals, they all seem to have a heavy Python footprint. My day-to-day is more about no-code/low-code SaaS tools and API integrations (Zapier, Make, etc.).
So my question is: can I realistically use these evaluation frameworks without diving deep into Python? Or is learning Python a non-negotiable prerequisite? I’m happy to write some configs or use YAML, but I’d prefer to avoid writing a lot of code from scratch.
If Python is a must, are there any particular resources or libraries that make the learning curve smoother for this specific use case? Conversely, if there are good GUI-based or API-only tools that let me run evaluations, I’d love to hear about those too.
Thanks in advance for any guidance.
– John
Always testing.
You can probably get started without writing a ton of Python, but you're going to hit a wall quickly. These frameworks are built by and for developers. The YAML/config options are usually just a thin wrapper over Python classes.
For your chatbot comparison, you might get further by treating the evaluation as a simple data pipeline. You can use a no-code tool to collect your LLM outputs into a CSV or a table, then use the evaluation library's API via a single Python script you run once. You won't need to master the language, just enough to call a function and pass a file path.
Look at UpTrain's Quickstart or LangSmith's hosted platform - they have UI components for comparing runs. That might be your best bet to avoid code altogether, but you'll pay for it and lose flexibility.
garbage in, garbage out