Alright, so I've been trying to orchestrate a simple research crew using CrewAI. The task is straightforward: analyze a provided company webpage (context chunk) and output a summary of their product and target market.
I feed it a clean, plain-text scrape of a startup's "About Us" page. No other links, no browsing, just the context.
And yet, I keep getting back confidently written paragraphs stating this B2B SaaS company is "publicly traded on the NASDAQ since 2015" and has "a primary customer base of retail consumers in the automotive sector." None of that is in the source material. Not even close. It's like the agents are pattern-matching from their training data and just... making up a company biography.
I've tried tweaking the `role` and `goal` to be hyper-specific, like "Analyst who ONLY uses the provided context. If information is not present, state 'Not specified.'" I've even played with lowering the `temperature` in the LLM settings passed to the agent. The hallucinations are less grandiose but still present—they'll invent revenue figures or partner names.
Is this a context window issue where the agent isn't properly attending to the full provided text? Or is the "creativity" of the underlying LLM bleeding through the CrewAI abstraction in a way that's hard to pin down? My A/B testing instincts want to isolate the variable, but with the layers of orchestration, it's tricky.
Anyone else running into their crews just... confidently fabricating details? What’s the fix—more aggressive prompt engineering in the agent definition, or is this a deeper workflow/config problem?
Data over dogma.