We've been integrating Claw's LLM API into our dbt project for generating dynamic documentation and field-level descriptions. Over the past two weeks, our scheduled jobs running between 10 AM and 2 PM EST have shown a marked increase in failure rates.
Our monitoring shows the pattern clearly:
- Average response time jumps from ~450ms to over 5 seconds.
- HTTP 429 (Too Many Requests) errors occur despite being well within our documented rate limits.
- Occasional 502 errors from their load balancers.
We built a simple retry logic with exponential backoff, but it's only a partial mitigation. The core issue appears to be service degradation under load.
```python
# Simplified version of our retry wrapper
def call_claw_with_retry(prompt, max_retries=3):
for attempt in range(max_retries):
try:
response = requests.post(CLAW_URL, json=prompt, timeout=10)
response.raise_for_status()
return response.json()
except requests.exceptions.Timeout:
if attempt == max_retries - 1:
raise
time.sleep(2 ** attempt + random.uniform(0, 1))
```
This is becoming a bottleneck for our data catalog automation. We're now considering:
* Implementing a circuit breaker pattern to fail fast and switch to cached descriptions.
* Shifting all non-critical Claw calls to a nightly batch window.
Has anyone else performed similar reliability benchmarking? I'm particularly interested in whether this is a regional endpoint issue or a broader infrastructure scaling problem. If this is a consistent trend, it forces a re-evaluation of building critical path processes on their API, regardless of the quality of their models.
Yeah, we saw the exact same pattern last quarter. The 429s despite being under the stated limits were the most frustrating part. Our workaround, which might help you, was to shift our non-critical jobs out of that 10-2 EST window entirely. We moved our documentation generation to run overnight.
We also found their rate limiting seems to apply per-region endpoint. If you're only using their default US endpoint, you might try explicitly routing a percentage of your calls to their EU one, if your data governance allows it. It helped spread the load a bit.
It does feel like a capacity issue on their end, unfortunately. Have you opened a ticket with their support? Sometimes highlighting a specific use-case like dbt automation gets their engineering team's attention faster.
Shifting non-critical loads is a band-aid, not a fix. It only works until everyone else starts doing the same thing and you've just moved your peak.
The regional endpoint suggestion is more interesting, but introduces its own latency and potential data sovereignty headaches. If you go that route, you need to make it a true weighted random distribution in your client, not just a static split, otherwise you'll just create a secondary peak in the EU. And you've now doubled your monitoring surface.
Their support tickets are a black hole. The real move is to start logging every single API call's timestamp, endpoint, response time, and HTTP code to your own metrics. When your own graphs show a clear, multi-week SLO violation *and* you can tie it to a tangible business cost (like delayed pipeline runs), that's when you escalate with their sales engineer, not support. Money talks.
Yeah, that timeout value in your retry wrapper might be part of the pain. A 10-second timeout is pretty aggressive when you're already seeing 5-second responses and getting load balancer 502s. You could easily burn through all your retries just waiting for timeouts during peak, never even hitting your backoff logic.
You might try dropping the timeout for the initial call down to something like 3 seconds during those peak windows. It sounds counterintuitive, but failing fast lets your backoff kick in sooner and actually spread the retries out, which is what you want. You can keep a longer timeout for the final retry attempt.
It's annoying to have to tune client logic around their instability, but it's kept our stuff running.
spreadsheet ninja
The "fail fast" strategy is sound for exactly the reason you state: it activates the backoff logic. We implemented a similar pattern.
A key addition: we made the timeout window dynamic based on recent percentile latency. If the p95 over the last hour is >4s, we drop the initial call timeout to 2s. It's a more adaptive hedge than a fixed schedule.
One caveat: if your final retry uses a much longer timeout (like 10s), you risk a single call monopolizing a worker thread during peak chaos. We cap all attempts at 4s and accept the higher hard failure rate - it's preferable to queue backup.
EXPLAIN ANALYZE