We've been integrating ContentBot's API into our documentation pipeline for generating release notes and service descriptions. Over the past week, our CI/CD jobs have started hitting consistent 429 (Too Many Requests) errors, causing several deployments to stall.
Our setup is fairly standard:
* A dedicated Kubernetes service account for the pipeline.
* Requests are batched, with no more than 5 concurrent calls from our side.
* We're using exponential backoff with jitter in our client, patterned after our service mesh retry logic.
Here's a simplified version of our retry logic:
```python
retry_strategy = Retry(
total=5,
backoff_factor=0.5,
status_forcelist=[429, 500, 502, 503, 504]
)
adapter = HTTPAdapter(max_retries=retry_strategy)
```
Even with this, we're seeing failures. The `Retry-After` header isn't always present, making graceful handling tricky.
The published rate limits in their documentation seem reasonable on the surface, but the practical enforcement appears very strict. Has anyone else run into this, particularly when triggering from automated systems? I'm trying to determine if:
* The limits are aggregate per account, rather than per key.
* There are hidden burst limits or minute-by-minute windows that aren't documented.
* We need to implement a client-side token bucket or similar to stay well under their thresholds.
Sharing your observations or any workarounds (besides just increasing delays) would be helpful. If this is a common issue, it might be worth the ContentBot team revisiting their throttling implementation or providing clearer telemetry.
That `backoff_factor=0.5` might be part of your problem. In urllib3/requests, that factor uses a base of `{backoff factor} * (2 ** (retry number - 1))` seconds. So your first retry is only 0.5 seconds, second is 1.0, third 2.0, etc. That's likely too aggressive if you're hitting a strict per-minute or per-hour limit.
I'd bump the `backoff_factor` much higher, maybe to 10 or 15, and definitely implement some client-side request throttling between your own batch calls. Your 5 concurrent calls might be fine, but if they're firing every few seconds from a pipeline, you're still hammering their global account limit.
Also, check if they count *all* HTTP calls, even failed ones, against the rate limit. Some APIs do, which makes retries a double-edged sword.
Pipeline Pilot
The backoff_factor point is good. Your retry logic, while well-intentioned, could be creating a self-inflicted surge against their limit if they're counting all HTTP calls.
You mentioned the Retry-After header isn't always present. This is a key detail. When an API vendor provides strict limits but doesn't consistently offer that header for automated handling, it shifts the burden entirely to the client and makes reliable integration difficult.
Before tweaking your code further, have you reached out to their support to clarify whether limits are per-account or per-key, and if they count failed requests? Your last point about trying to determine that is exactly where you should focus next. Sometimes the published docs don't match the practical reality of their enforcement system.
Keep it constructive.
Your math on the backoff factor is correct for the default behavior. However, it's worth checking if their `Retry` class has been configured with `backoff_max`. If it hasn't, that exponential growth could still be capped too low for a per-hour limit, even with a higher factor. I've seen cases where people set a large factor but also set `backoff_max=60`, which defeats the purpose entirely.
The more critical point you raised is about counting all HTTP calls. This is a huge gotcha. If their enforcement layer counts every 429 response as a request, then a tight retry loop just accelerates the lockout. You'd need to track your own client-side quota, decrementing only on a successful 2xx response, which adds significant complexity.
-- bb42