We just finished a pilot using Relevance AI's vector search and workflows for a legal document retrieval app. The goal was to let 50 internal users search across 100k+ past case files and contracts using natural language. The initial prototype was magical, but scaling to the team exposed some serious cracks in the foundation.
Here’s what broke under real, concurrent usage:
**1. The Webhook "Guaranteed Delivery" Wasn't.**
We built a workflow that triggered a webhook to our internal system when a new document cluster was identified. Under light testing, it was fine. At scale, we saw about a 5% failure rate on webhook calls. The logs showed timeouts, but no built-in retry logic from Relevance. We had to build a queuing system on our end.
```json
// The webhook payload was solid, but the delivery...
{
"workflow_id": "cluster_alert",
"documents": [...],
"status": "completed"
}
// ...would just sometimes never arrive.
```
**2. API Rate Limits Hit Without Clear Headroom.**
The 50 users, all searching simultaneously during peak hours, triggered rate limit errors (429s) on their API. The docs state limits, but not how they're calculated across different endpoints (search vs. workflow execution). It created a terrible user experience where searches just failed silently in the UI.
**3. JSON Parsing Gremlins in Complex Outputs.**
When the AI workflow returned large sets of retrieved documents with metadata, the JSON structure for "reasoning steps" would occasionally be malformed—unclosed brackets or escaped quote issues. It broke our parsing logic downstream. We had to add a strict validation and cleanup layer in Make before sending data to our app.
**4. Zapier/Make Connector Limitations.**
We used Make for orchestration. The native Relevance AI module only supports the basic "Run a Workflow" action. For error handling, checking status, or managing the async jobs, we had to drop to the HTTP module and handle all the auth and polling ourselves. It defeated the purpose of a managed connector.
On the plus side, the relevance of search results was consistently high, and the workflow builder for chaining search → summarize → categorize is visually intuitive. But the platform feels like it's in a prototype stage for anything beyond a solo developer's project.
Has anyone else hit these scaling walls? Specifically:
* How did you handle webhook reliability?
* Is there a way to monitor or predict rate limit consumption per user?
chloe
Webhooks or bust.