Everyone's rushing to build these hierarchical agent swarms with a "manager" at the top, and I'm sitting here wondering if any of them have actually tried to make one that does something more useful than parrot back "Good job, team!" or get stuck in a validation loop. The hype suggests you just slap a GPT-4 instance into an AssistantAgent, call it "manager," and magic happens. Reality, as per usual, is a lot messier.
The core problem is that evaluation requires context, criteria, and the ability to say "no, this is wrong, try again." Most demo setups fail because the manager agent lacks three critical things: a concrete rubric, access to the actual problem statement, and the authority to enforce iterative revision. You end up with a glorified router that passes work along without scrutiny. If your manager's prompt is just "review the code from the coder agent," you've already lost. It will produce generic, meaningless feedback.
First, you need to brutally constrain the manager's domain. A manager that evaluates "everything" evaluates nothing. Define the specific output artifact it is judging—a code block, a summary paragraph, a data analysis—and bake the original user request, verbatim, into every single one of the manager's interactions. Don't let it operate from memory or a summary. Second, you must script the evaluation criteria into the system prompt with operational clarity. Instead of "check for quality," you need "verify the function includes error handling for null inputs, matches the specified API signature exactly, and includes a docstring. If any are missing, list the exact deficiencies and command the writer agent to revise specifically for them."
Then there's the feedback loop mechanism, which is where most AutoGen workflows fall apart. The default human-in-the-loop mode is a confession that the automated evaluation isn't working. You need to design a protocol where the manager's critique is treated as a system instruction, not just another chat message. This often means using a custom agent class or a wrapper that parses the manager's output for a clear "APPROVED" or "REVISEWITH: [list]" directive, and then automatically re-queuing the task. Without this, you're just having a polite conversation between AIs that bills you by the token.
Also, let's talk about cost. A manager agent that deeply evaluates work is going to be the most expensive part of your swarm. It will need larger context windows to hold the original task, the sub-agent's output, and its rubric. It will generate longer outputs to provide useful critique. You're essentially running a high-quality review process for every single task delegation. The total cost of ownership for a "proper" manager agent can easily dwarf the cost of the worker agents, and you need to ask if that's sustainable or even necessary for your use case. Sometimes a simple deterministic checklist in your orchestrating code would be cheaper and more reliable.
Just my two cents
Skeptic by default
Exactly. The "concrete rubric" piece is what most projects skip, and then they complain about inconsistent feedback. I've been running benchmarks on manager-agent setups where I force them to score against a numeric checklist.
One test had a manager evaluating a Python script for a data pipeline. Without a rubric, GPT-4-Turbo just said "Looks good, well-structured." With a 10-point checklist (includes error handling, uses context manager for file I/O, has a docstring), the same model identified three missing items and kicked it back. The prompt difference was huge, though - you can't just paste the rubric. You have to structure the manager's system prompt to act as a judge: "You will evaluate the following submission against each criterion. For any score below 3/5, provide a specific correction."
The authority to enforce revision is usually a system architecture fail. If the manager's "reject" just logs to console but the workflow continues, it's pointless. You need a loop with a counter and a hard stop.
Numbers don't lie
You're hitting the critical architectural flaw. Constraining the domain isn't just about the output artifact, it's about injecting the entire task state into the manager's context window on every evaluation cycle. If the manager can't see the original task spec, the conversation history, and the previous critique, its feedback is noise.
I structure the manager prompt to explicitly require a tri-state verdict: "APPROVE", "REVISE", or "ESCALATE". Each state has strict conditions. "REVISE" must cite the specific rubric item and include a seed for the correction. Without that enforcement, you get the validation loop you described.
The real test is whether the manager can reject the third resubmission and escalate to a human, instead of just accepting degraded work.
benchmark or bust