Just spun up a new custom tool in the Relevance AI dashboard. The configuration saved, the UI gave me a green check, and yet when I try to call it from an agent, I get a persistent 'Tool not found' error. The immediate, almost instinctual, response is to refresh the page or clear the cache—a modern incantation that somehow fixes half the web.
But in a platform built for orchestrating automated workflows, shouldn't tool availability be consistent and immediate? I’ve seen similar eventual consistency issues in other API-driven platforms, and it’s always a red flag for underlying state management problems. It makes me wonder about the actual architecture. Is there a propagation delay between the control plane and the execution environment? If so, that’s a concerning gap for anything requiring real-time tool creation or modification.
Has anyone else hit this wall? What was the actual resolution—did you just wait it out, or was there a specific refresh mechanism beyond the browser? More importantly, has Relevance AI documented this latency or considered it a ‘feature’ rather than a bug? For a security-minded user, undocumented state delays are a great way to introduce unpredictable behavior during audits or compliance checks.
—Greg
Trust but verify
That propagation delay is almost definitely what's happening. I've run into similar patterns with other API-first platforms where the management API and the execution runtime don't share the same datastore or cache layer. The UI gets its confirmation from the control plane API, but the agent is querying a separate tool registry that updates on a slower sync loop.
You asked if waiting it out works. In my experience, yes, usually a few minutes. But that's not an acceptable fix if you're automating anything. The workaround I've had to use elsewhere is to add a retry or a sleep step in the automation script that creates the tool before immediately trying to use it. It's a band-aid.
The real question is whether they expose an API endpoint to trigger a sync or check the tool's operational status. If they don't, you're stuck guessing at the delay, which is pretty amateur for a workflow platform. Check their docs for something like a 'tool sync' or 'refresh agent tools' call. If it's not there, file a ticket and call it out as an operational risk.
Automate everything. Twice.
Yep, that 'band-aid' retry loop is exactly where my mind goes too. It's a pattern that feels hacky but ends up being a permanent fixture because the underlying sync problem is rarely treated as a P0 bug. It's the classic decoupling side-effect.
Have you ever had to implement a backoff strategy for something like this? I've set up exponential backoff in a script before, but it always feels like over-engineering a workaround for what should just be a synchronous API call.
And you're spot on about checking for a sync endpoint. If they don't have one, the next hack is trying to 'touch' the tool by making a dummy update via the API, hoping it flushes the change faster. Not great.
editor is my home