Your `backoff_factor=0.5` is the main problem, but the real issue is that your retry logic treats a 429 like any other server error. It isn't. A 429 is a directive from their system saying "you are over quota *right now*," not "something's broken, try again soon." Retrying with a half-second delay is just queuing up more requests against a locked door.
The lack of a `Retry-After` header is a major API design flaw on their part, but you have to work around it. You need to treat the first 429 as a signal to stop, not to start a retry loop. Implement a circuit breaker that halts all calls from that service for a configurable period after hitting a rate limit, instead of letting each worker independently retry. That centralizes the backoff state and prevents your five concurrent calls from turning into 25 rapid-fire retries.
APIs are not magic.
The point about your retry logic treating 429s like server errors is spot on. It's a policy error, not a system failure.
While you're waiting for clarification from ContentBot support, you could implement a simple memory in your service to share the 'throttled' state across your pods. A short lived value in a shared cache, like Redis, could act as a makeshift circuit breaker to coordinate a pause after the first 429 hits. This prevents the other four concurrent calls from immediately retrying and making the situation worse.
That said, if the limit is truly per-account and you're sharing it with other teams, no amount of client side logic will fully solve it. Have you been able to isolate if the errors spike at a particular time that aligns with another team's schedule?
—HR
The shared Redis cache for a circuit breaker is a clever tactical fix, I've done that before. One gotcha is making sure the TTL on that 'throttled' key is long enough, but not too long. You don't want it to expire mid-window and unleash another barrage.
> if the limit is truly per-account and you're sharing it with other teams, no amount of client side logic will fully solve it.
This is the real heart of it. A shared cache might stop your own service's pods from stampeding, but it's silent to everyone else using the same master account. Until they clarify the enforcement boundary, you're basically tuning in the dark. Did you get any traction with their support on that per-account vs. per-key distinction?
Data nerd out
That's a critical detail about the `status_forcelist`! I almost got burned by that last year with a different SDK. Their docs advertised automatic retry for "common errors," but 429 wasn't on the default list. You had to explicitly add it.
Your point about the 60-second sleep being diagnostic is perfect. I'd add that if the sleep *doesn't* work, it tells you the limit might be on a rolling window or a sliding algorithm, which is trickier to pinpoint.