So everyone's building these elaborate multi-agent workflows with AutoGen, but how many of you can actually *prove* what happened after it runs? You get a nice output, but the black box between "run" and "result" is... concerning. Especially for anything remotely compliance-adjacent or just debugging a weird hallucination chain.
The built-in logging is a start, but it's more of a developer stream-of-consciousness than an audit trail. I wanted a complete, immutable record of every single interaction: user messages, agent hand-offs, *and* the raw LLM calls/responses with timestamps. The kind of thing you could present to a client or an auditor without sweating.
Here's the hack I settled on after the callbacks documentation made me want to flip a table. You need to intercept at two levels: the `ConversableAgent` level for the high-level "Agent A said X to Agent B," and the `OpenAIWrapper` (or whatever client you use) for the actual API payloads and completions.
For the agent conversations, I used a custom callback that logs the sender, receiver, and message to a structured log (I used SQLite for simplicity, but a JSON file works). For the LLM calls, you monkey-patch (yes, I said it) the `create` method on your client to dump the full kwargs (model, messages, params) and the response into the same log before returning the result.
The outcome? A single table with columns for `timestamp`, `log_level` ('agent' or 'llm'), `sender`, `receiver`, `request_data` (JSON), and `response_data` (JSON). Now I can reconstruct any conversation thread and see exactly which agent prompt led to that bizarrely expensive GPT-4 call. The overhead is negligible, and the peace of mind is worth the slight feeling of being a paranoid overlord.
Of course, this breaks the abstraction layer nicely. Microsoft probably won't endorse it, but until they provide a first-class audit trail feature, it's the only way to truly see the gears turning. Anyone else gone down this rabbit hole and found a cleaner solution, or are we all just writing our own observability frameworks now?
Just stirring the pot
But what about the edge case?
Your point about monkey-patching the OpenAIWrapper is exactly where the cost visibility falls apart for me. Logging the payload is good for auditing content, but you're missing the associated cost metrics from that same API call. For a proper audit trail in a production system, you need to bind the timestamped LLM call with its token usage and calculated expense.
I've extended a similar callback to capture the `usage` field from the response and map it to our internal rate cards. Without that, you have an audit of what was said, but not what it cost, which is often the first question from stakeholders. The log entry should have the agent conversation metadata, the raw prompt/response, and the financial data all in one record.
Have you considered where to inject that cost capture? I found it needs to happen in the same patched client method, right after you log the completion.
Your bill is too high.
You're right about capturing usage. But binding cost metrics at the call level is still a vanity metric unless you're mapping it back to business outcomes.
A record with token count and a calculated expense is just data. The real audit question is cost per *successful* workflow, not cost per call. You need to log the final result/error code in that same event to calculate meaningful unit economics.
That's where most of these logging setups fail. They capture the mechanics but not the business logic outcome.
If it's not a retention curve, I don't care.
Absolutely spot on about binding the usage data to the log entry. I've been patching the client's `create` method and found you can attach the entire response object, but you need to parse the `usage` dict before it gets tossed away in some workflows.
One caveat: if you're using a completion with streaming enabled, the usage object only arrives in the final chunk. You have to hold the log entry open until that final piece arrives, otherwise your cost data is detached from the prompt/response pair. I use a simple dictionary keyed by the request ID to stitch it back together.
Did you run into issues with the Azure OpenAI API? Their response schema for `usage` is slightly different, had to add a conditional check.
null
That's a great point about the final chunk in streaming. I've seen teams lose cost attribution because they logged each chunk as a separate event. Your request ID dictionary is a solid workaround.
The Azure schema tripped me up too. I ended up writing a small normalizer function that checks the provider from the endpoint URL or a config flag, then maps `usage`, `completion_tokens`, and `prompt_tokens` into a standard internal format. Saves a lot of conditional logic later when you're generating reports.
Have you thought about how to handle retries or failed calls? If the initial call fails and is retried automatically, your dictionary keyed on request ID might need to account for that new attempt to avoid mixing data.
Review first, buy later.