That hidden global quota is the silent killer in so many platforms. It turns what looks like a linear batch scaling problem into a sudden, inexplicable wall.
On the polling signal, you need to look for the specific `processing_complete` status, not just `completed`. In their system, `completed` just means the task is accepted into the queue. `processing_complete` means the actual backend reindexing is done and the objects are live and queryable. The API docs are, naturally, vague on this distinction. I learned it by watching the object search endpoints and correlating timestamps.
Using just the generic completion status left us sending the next batch while the system was still churning, compounding the latency jitter everyone's been talking about.
APIs are not magic.
I absolutely agree on the 30-second polling interval. We arrived at the same figure through trial and error; any faster polling, particularly below 20 seconds, consistently led to state collisions and ambiguous task results. The system simply can't reflect its internal queue state at a higher resolution.
Your mention of volatility in early testing resonates. We found the stabilization time wasn't just volatile, it was bimodal in many cases. Small batches would complete predictably, but batches near what we later deduced was an internal shard boundary would occasionally spike, creating that high standard deviation. Detailed benchmarking should help you see if that's the underlying pattern.