Hi everyone, total beginner here trying to solve a real problem at my small company. 😅
We have a huge internal WordPress wiki (like, thousands of pages). I'm trying to set up a simple Q&A system so our team can ask natural questions and get answers from our docs. I'm stuck between using a CrewAI agent or just a custom GPT.
My main worries are cost and accuracy. I'm nervous about setting up a CrewAI flow that might get expensive if it's doing tons of document reading for every query. But I also worry a GPT might hallucinate too much and give wrong answers about our internal processes.
Has anyone tried something similar? I'd be so grateful for any real-world thoughts on which path might be more predictable for a fixed monthly budget, and which one actually sticks to the source material better.
I run infras for a 120-person consulting firm. We migrated from Confluence to a static wiki last year and built a Q&A system. We tried CrewAI on GCP and Azure OpenAI before landing on something else.
- **Accuracy:** CrewAI can be more accurate *if* you tune it perfectly, but that's a big if. It uses retrieval to pull doc snippets before answering. In our tests, GPT-4 with a good vector store matched it for factual answers, but both hallucinated on ambiguous questions. The real difference was maybe 5% more accuracy with a finely tuned CrewAI flow, not worth the hassle.
- **Cost Structure:** GPTs (Azure or OpenAI) are stupidly simple: tokens in/out. You can predict cost per query. A CrewAI agent doing retrieval and calling multiple LLM steps multiplies that. Our early prototype ran a query chain that cost ~$0.12 per ask because it read so many chunks. Azure OpenAI was ~$0.03 for similar quality.
- **Deployment Effort:** CrewAI means building and hosting a pipeline - containers, queues, maybe a dashboard. That's weeks. A custom GPT is an afternoon. Even using the API with a prebuilt vector store is a weekend project.
- **Where CrewAI Breaks:** It gets expensive fast if you don't cap retrieval steps. We saw loops where it kept searching for "more context" and racked up 10+ LLM calls per query. Also, you're on the hook for monitoring and scaling the infra.
I'd use a custom GPT or the Assistants API with a RAG setup. It's predictable and good enough. The real question is your monthly budget and whether you have DevOps time to burn. Tell us your budget and if you have a dev who can own a container setup.
show the math