I've been conducting an extensive evaluation of AgentGPT over the past several weeks, primarily using a self-hosted instance via Docker to maintain data sovereignty, but I have also tested the cloud offering for comparative analysis. A persistent and frankly puzzling issue has emerged: the output formatting from my agents is remarkably inconsistent, to the point of compromising automated workflows. This isn't merely a cosmetic concern; it directly impacts the ability to parse and utilize the agent's results in a downstream, programmatic manner.
The core of the problem appears to be a lack of deterministic behavior in how the agent structures its final answers. For identical tasks, with identical initial prompts and context, I receive outputs in vastly different formats. Consider this simple task I've run multiple times: "Compile a list of the top 5 open-source backup solutions, with their primary feature and license type."
Here are two actual outputs from separate runs:
**Run 1 Output (Structured):**
```
1. **BorgBackup**
- Primary Feature: Deduplication, compression, encryption.
- License: BSD-3-Clause.
2. **Restic**
- Primary Feature: Deduplication, backends to various storage types.
- License: BSD-2-Clause.
... (and so on)
```
**Run 2 Output (Unstructured Paragraph):**
```
Certainly. Top open-source backup solutions include BorgBackup which offers deduplication and compression under a BSD-3-Clause license. Restic is another option with deduplication and support for multiple backends, licensed under BSD-2-Clause. Other notable tools are Duplicati, which has cloud storage support and is under GPL, and Kopia, with snapshot policies and an Apache License. Finally, Burp is a network backup system under AGPL.
```
My environment configuration is stable:
```yaml
# docker-compose.yml relevant section
agentgpt:
image: agentgpt/agentgpt:latest
environment:
- OPENAI_API_KEY=${OPENAI_API_KEY}
- DEFAULT_MODEL=gpt-4-turbo-preview
- MAX_LOOPS=50
volumes:
- ./data:/app/data
```
I have observed this phenomenon across different models (`gpt-3.5-turbo`, `gpt-4`) and with varying complexity of tasks. The inconsistency manifests in:
- The presence or absence of Markdown headers (`##`, `###`).
- The use of bullet points versus numbered lists versus plain paragraphs.
- The inclusion of introductory or concluding sentences, which seems stochastic.
- Indentation and code block formatting for non-code answers.
This suggests the issue may lie in the agent's "post-processing" logic, or perhaps in the fundamental system prompt that governs the final output stage. It seems the agent does not have a enforced, final "output formatting" phase, leaving the underlying LLM too much latitude.
Has anyone else in the community encountered this level of inconsistency? More importantly, has anyone devised a strategy—perhaps through a custom prompt injection in the agent configuration or by modifying the self-hosted codebase—to enforce a strict output schema or template? I am considering implementing a secondary parsing agent to normalize outputs, but this seems an inefficient solution for what should be a core reliability feature of an autonomous agent platform.
Take back control