I've been evaluating AutoGen for orchestrating a multi-agent code review workflow, and I've consistently run into a problem where agent responses are truncated mid-sentence, often at what seems like a predictable length. This is a classic symptom of hitting a context window limit, but the specifics in AutoGen require a bit of digging. The issue isn't always with the primary LLM's own token limit, but with how AutoGen manages the conversation history it passes into each call.
From my analysis, the truncation typically occurs because the `max_consecutive_auto_reply` setting in a `ConversableAgent` is working in tandem with the accumulated token count of the entire message history. When the total history exceeds the model's context window, the responses from the LLM get cut off, as the model is trying to complete its thought within the remaining tokens. You might be seeing clean cuts or garbled output at the end.
To diagnose, you need to examine both your agent configuration and the underlying LLM config. Here's a simplified version of the setup I was troubleshooting:
```python
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
config_list = config_list_from_json(
"OAI_CONFIG_LIST",
filter_dict={
"model": ["gpt-4-turbo-preview"],
},
)
llm_config = {
"config_list": config_list,
"max_tokens": 4096, # This is the *response* token limit, not the context window.
"temperature": 0,
}
assistant = AssistantAgent(
name="code_reviewer",
system_message="You are a senior software engineer.",
llm_config=llm_config,
max_consecutive_auto_reply=10,
)
user_proxy = UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
max_consecutive_auto_reply=5,
code_execution_config=False,
)
```
The key points to verify are:
* **Underlying Model Context Window:** Confirm the actual context window of the model you're using (e.g., `gpt-4-turbo-preview` has 128k, but `gpt-3.5-turbo` has 16k). The `max_tokens` parameter in the `llm_config` only controls the maximum number of tokens the model can generate in its response, not the total tokens it can receive.
* **Accumulating History:** Each turn in the conversation adds the previous messages to the prompt. A long-running multi-turn discussion between agents can quickly bloat the context. AutoGen does not automatically summarize or prune old messages by default.
* **Termination Triggers:** A `max_consecutive_auto_reply` limit will stop the loop, but truncation *within* a single reply is a token limit issue.
The mitigation strategies depend on your workflow:
1. **Implement a Summarizer Agent:** For long-running chats, introduce a dedicated agent that periodically summarizes the conversation and replaces the history with a condensed version.
2. **Use `summarizer_llm_config`:** Some Agent classes accept a separate LLM config for a lighter, faster model to handle summary tasks.
3. **Limit Conversation Depth:** Reduce `max_consecutive_auto_reply` and design your agent interactions to be more atomic, clearing the history between distinct tasks.
4. **Switch to a Larger Context Model:** If cost permits, moving from a 16k to a 128k model can be a straightforward, if expensive, solution.
5. **Monitor Token Usage:** Enable logging or use the LLM's usage metadata (if provided by the wrapper) to track the prompt token count before the cut-off occurs.
In my specific migration scenario, the truncation was breaking a JSON output that an agent was supposed to generate. The solution was to implement a custom termination function that checked for a valid JSON bracket closure and, if truncated, would trigger a summarization step before retrying the request. The root cause was indeed the 16k context window being filled after ~15 rounds of detailed code analysis.
Yeah, the interaction between `max_consecutive_auto_reply` and the unmanaged message history is the core of it. Your point about the LLM trying to finish a thought within the remaining tokens explains why the truncation often looks syntactically plausible, not just a raw cut.
One nuance I've seen is that the cost accumulates faster with function calls, because the serialized output of the tool/function gets appended to the history in full. If your code review agents are calling linters or security scanners, that can bloat the context much quicker than plain text replies.
Have you looked at enabling `human_input_mode="ALWAYS"` as a debug step? It forces a pause and clears the consecutive reply counter, which sometimes reveals if the truncation is purely length-based or if there's a prompt-level issue.
Your diagnosis is solid for the symptom, but it's incomplete if you stop at configuration inspection. You've correctly identified the interplay between `max_consecutive_auto_reply` and token accumulation. However, focusing solely on the LLM config or agent settings misses the architectural risk in long-running multi-agent chats.
The real culprit is often the lack of a summarization or truncation strategy *between* turns. AutoGen's default behavior is to just keep appending. In a code review flow, with multiple agents and function outputs, you're essentially building a single, unbounded document. Even with a generous context window, you'll eventually hit a wall.
Have you considered implementing a custom `message_trim` handler to prune the history proactively, rather than letting the LLM fail silently with a cut-off? The truncation you see is the failure mode, not the problem itself.
James K.