Just tried to run a batch process of about 500 prompts through the Claude API. Spent more time handling 408 and 504 errors than actually getting results. The promised "high throughput" seems to vanish once you push beyond a trivial number of requests.
My setup is straightforward. Python, using `asyncio` with semaphore to manage concurrency, exponential backoff on retries. Works flawlessly with other endpoints. With Claude, it's a coin toss whether a job over 100 items completes without intervention.
```python
# The retry logic that shouldn't be necessary
async def send_with_retry(session, payload, max_retries=5):
for attempt in range(max_retries):
try:
async with session.post(ENDPOINT, json=payload, timeout=30) as resp:
if resp.status == 408 or resp.status == 504:
raise asyncio.TimeoutError(f"HTTP {resp.status}")
return await resp.json()
except (asyncio.TimeoutError, ClientConnectorError):
if attempt == max_retries - 1:
raise
await asyncio.sleep(2 ** attempt)
```
Is this just their infrastructure buckling under load, or is there some undocumented rate limiting per minute that's more aggressive than they state? The console metrics don't show any quota breaches. It feels like being throttled for simply using the service as advertised for batch processing.
For the price, I expect reliability, not a guessing game. Makes the open-source models you can run on your own hardware look more appealing by the minute, even if they're a few percentage points "dumber" on benchmarks.
Just my 2 cents
Just my 2 cents