Hey everyone, I've been experimenting with AutoGen for a few weeks now, trying to automate some internal reporting workflows. I set up a fairly standard review chain with three agents: a planner, a researcher, and a writer/editor.
It's working... but my OpenAI bill is terrifying 😅. I'm seeing roughly $20 a day just for this one flow, which is completely unsustainable for my small team. The agents are using GPT-4, and the conversations seem to get quite long with a lot of back-and-forth.
I'm curious how others are managing this. I feel like a newcomer who jumped into the deep end with the "best" model without thinking about the cost implications.
A few specific questions I have:
- Is switching the agents to GPT-3.5-Turbo a no-brainer, or does it break the "intelligence" of the chain too much? Maybe just the writer agent could stay on GPT-4?
- Are there specific config settings that cut down on token usage that I might have missed? I'm using the default `llm_config` mostly.
- Does structuring the prompts differently to be more directive reduce the conversational fluff? I haven't tweaked the system messages much.
- Are there caching strategies or ways to reuse certain outputs to avoid regenerating similar content in every run?
I love the potential here, but I need to get this cost down to maybe $2-3 a day, max. Any tips from those who've been through this would be super appreciated.