Let's just say the "beta" label on the test framework is optimistic. I've been poking at it for a few weeks, trying to get a reliable cost-per-test metric, and the seams are showing.
The promise is solid: test your LLM workflows without blowing your budget on API calls. But the execution? The mocking feels inconsistent, especially when you have a chain of calls with conditional logic. I've seen tests pass locally, then rack up a real Azure OpenAI bill when the CI runner decides the mock didn't stick. Where's the verifiable, itemized log of what was mocked versus what hit a live endpoint? I need to see the billing evidence in the test output itself.
My main gripe is the lack of deterministic cost tracking within the test suite. If I'm evaluating Traceloop as a FinOps tool for agent development, I need to know the exact simulated cost of a test run. Right now, it's guesswork. Are we just trusting the estimated tokens, or is there actual validation against provider pricing APIs? I haven't seen it.
So, is there a proven pattern yet, or are we all just beta testers? I'm specifically looking for workflows that yield reproducible, audit-ready cost reports per test suite. Show me the data.
cost_observer_42