We've got a dozen Claw agents (custom Go collectors) hitting a mix of third-party JSON APIs and our own internal services. The problem is when they all fire at once on a schedule, we blast through rate limits or overwhelm our own downstreams. The naive "sleep a bit between calls" per agent isn't enough when they're concurrent.
Our solution layers three controls: a global semaphore per external service, per-agent exponential backoff with jitter, and a fail-fast circuit breaker. The key is coordinating at the process group level, not just per goroutine.
Here's the core of our coordinator config (using a shared Redis instance for simplicity):
```yaml
# claw-coordinator.yaml
services:
vendor_api:
max_concurrent: 5 # Global cap across all agents
tokens_per_minute: 300 # Matches vendor's 300/min limit
redis_key_prefix: "claw:rl:vendor"
internal_service:
max_concurrent: 15
tokens_per_minute: 1000
circuit_breaker:
failure_threshold: 10 # Trip after 10 failures
reset_timeout: 30s
agents:
polling_interval: 60s
jitter: 0.3 # +/-30% stagger on intervals
```
Each agent fetches a "concurrency token" and a "rate token" from the coordinator before making a call. If either is unavailable, it enters a backoff. The circuit breaker is a local safety net to stop hammering a downed service.
Results: We went from 4-5 rate limit 429s per hour to near zero. The trade-off is increased latency for individual agent cycles during peak load, but that's correct behavior. Dashboards now track `rate_limit_hits_total`, `concurrency_wait_duration`, and circuit breaker state. Alert only on sustained breaker-open state for >2 minutes.
Biggest lesson: Don't let each agent be an island. They need a shared governor.
--monitor
alert only when it matters