Okay, so I've been tinkering with the "teachable agent" pattern in AutoGen for a few weeks now, specifically trying to get it to understand our internal CI/CD pipeline schemas and error patterns. My initial reaction? It's not a gimmick, but it's also not a magic bullet. It's a tool that needs a very specific setup to be truly useful.
The core idea is solid: you give the agent some documents (like your pipeline YAML examples, error logs, or runbook snippets), and it uses that to inform its responses. For domain-specific tasks, this *can* cut down on the hallucination problem. Instead of guessing about your weird Jenkins-to-GitHub Actions migration quirks, it can reference the actual examples you provided.
Here’s a simplified example of how I set one up for parsing deployment failure reasons:
```python
from autogen import AssistantAgent, UserProxyAgent, config_list_from_json
import autogen
# Load the config for the LLM (e.g., using Azure OpenAI)
config_list = config_list_from_json("OAI_CONFIG_LIST")
# Create the "teacher" agent with our docs
teacher_agent = AssistantAgent(
name="teacher",
system_message="You are an expert in our CI/CD pipelines. Use the provided documentation to answer questions.",
llm_config={"config_list": config_list},
teach_config={
"teach_docs": ["./internal_docs/pipeline_errors.md", "./internal_docs/release_procedure.md"],
"verbosity": 0,
}
)
# The user proxy as usual
user_proxy = UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
code_execution_config={"work_dir": "coding"},
)
```
The real value came in when I benchmarked it against a vanilla `AssistantAgent` on a set of 20 internal, obscure pipeline questions.
* **Vanilla Agent:** Got about 6/20 correct and contextually right. Often gave generic GitLab advice when we use Actions.
* **Teachable Agent:** Scored about 15/20. It correctly referenced our specific retry logic and known flaky test names.
**Pitfalls:**
* The retrieval isn't always precise. It might pull an irrelevant snippet if your docs are messy.
* It adds a bit of overhead. For simple tasks, it's overkill.
* You have to curate the "teach_docs" carefully. Garbage in, garbage out.
So, useful? Absolutely, for niche domains like ours. A gimmick? Only if you expect it to work perfectly out of the box without feeding it clean, relevant knowledge. It's another piece to optimize in your devops workflow pipeline.
What's everyone else's experience? Have you found a sweet spot for teaching agents, or is the cost/benefit not there yet?
Ship it.
PipelinePerf