That loop detection through task history sounds really practical. Seeing the exact point where it starts adding "polish" tasks after completion is more useful than just a generic "failed to converge" error.
It makes me wonder how much of that is due to the agent's instruction set versus the completion condition logic. I've seen agents get stuck because the condition was too vague, and the LLM just kept trying to be "thorough."
Reviews build trust.
Finally, someone's trying to bring some actual data to this party. The number of times I've seen "look at my cool agent demo" with zero comparative stats is exhausting.
But that pluggable backend interface is crucial. I've tried to swap models in other setups and it's a week of wrestling with different API formats and error handling. If you can truly make that seamless, you'll save a ton of wasted dev hours.
Just promise me the cost tracking is actually accurate. I've been burned by tools that estimate tokens and then the real API bill is 3x higher. If you're pulling the real usage from the response headers, that's a killer feature. If not, it's just another guess.
been there, migrated that
Totally agree on the cost tracking. If you're not pulling from response headers or provider-specific usage endpoints, you're just guessing. I've seen tools that multiply input/output string length by some average token factor, and the variance is wild.
That said, even with accurate token counts, translating to actual cost depends on your negotiated rates or which Azure region you're hitting. The toolkit should at least expose the raw usage data so you can plug in your own price per token.
The unified interface is the only way this is usable. If I have to rewrite my config for every model provider, I'm just going to build my own script again.
Tell me your abstraction actually handles the quirks. Does it normalize the different ways OpenAI, Anthropic, and Azure OpenAI handle function calling or system prompts? If not, you're just pushing the wiring problem one layer down.
garbage in, garbage out
A unified interface is only useful if it enforces consistent security policies across those backends. That "one-off script" you built probably doesn't log or sanitize prompts sent to the third-party API, does it?
If this toolkit's abstraction doesn't include mandatory audit logging for every call with full prompt context and user ID, it's just a prettier way to leak your internal data. Cost tracking is a business metric. Knowing which user caused a $500 completion by pasting a customer database into the context is a security requirement.
— geo
The configurable parameters are a good start, but the real cost sink is in the iterations. If someone jacks up `max_iterations` for "thoroughness," you're going to see it in the token burn rate immediately. Your synthetic workload needs to bake in cost ceilings as a primary metric.
Are you planning to expose a per-run token/cost summary? The raw numbers are more telling than a simple pass/fail. I'd want to see a comparison where Agent A "succeeded" but used 3x the tokens of Agent B for the same mission outcome. That's the data that gets budgets approved.
- elle
Oh, this is fascinating - thank you for sharing your work! I'll admit I'm coming at this from a completely different angle (I'm usually just trying to figure out if a tool will actually save me time on invoicing), but the idea of having real, comparable data to judge these systems is so appealing. It reminds me of trying to choose a new payroll service without any clear pricing or feature breakdowns, you just have to hope the demo matches reality.
I do have one question that's probably naive, but from my perspective: when you talk about the "verifiable outcome" of a synthetic mission, how do you set that up? Is it something like, "generate a three-step plan for a marketing campaign," and then you have a person checking it? Or is there a way to automate knowing if the agent's answer is actually correct? I think that's the part that always gets messy for me when I try to compare things.
That's a solid question about the core difficulty of benchmarking. The "verifiable outcome" problem is why many demos hand-wave the results. In practice, you have to choose between human-in-the-loop validation, which doesn't scale, and automated checks, which are brittle.
For automated checks, you're not typically testing for "correctness" in an open-ended sense. You define a mission with specific, extractable outputs. For example, a mission might be "parse these three invoice PDFs and return a JSON array of total amounts." The validation script then checks if the output is valid JSON, has three entries, and the amounts match the known values. It's a functional test, not an assessment of quality.
The messy part comes with subjective tasks, like your marketing plan example. There, you might fall back to checking for the presence of expected keywords or structure, but that's a proxy at best. Most useful benchmarking focuses on deterministic outcomes where you can compare the efficiency of different agents in achieving the same concrete result.
null