We've been using Perplexity's API for a few months to handle summarization tasks in our CI/CD pipeline. The output quality and latency are solid for the price.
But their rate limiting is brutal and unpredictable. Our workflow batches processing after deployments, and we keep hitting 429s. It's stalling our entire automation.
* We're on the supposed "unlimited" Pro plan.
* Limits seem to be per-minute, but the docs are vague.
* Retry logic with exponential backoff helps, but it blows up our 95th percentile latency, making SLAs impossible.
Has anyone reverse-engineered their actual rate limit buckets or found a reliable configuration? We're considering a switch, but need comparable output quality.
Our current fallback logic looks like this:
```python
@retry(stop_max_attempt_number=5, wait_exponential_multiplier=1000)
def call_perplexity(prompt):
response = requests.post(
'https://api.perplexity.ai/chat/completions',
headers={'Authorization': f'Bearer {API_KEY}'},
json={"model": "sonar", "messages": [{"role": "user", "content": prompt}]},
timeout=30
)
if response.status_code == 429:
# This gets triggered too often
raise RateLimitError
return response
```
What are others using for high-throughput, low-latency summarization that won't throttle like this? Looking at OpenAI GPT-4 Turbo, Anthropic Claude Haiku, or maybe Cohere. Need hard numbers on:
* Actual sustained requests per minute (RPM) limits
* p95 latency under load
* Cost per 1k tokens for 500-1000 word summaries
Benchmarks or bust.
I've seen this pattern before with other "unlimited" API plans, especially when dealing with batch operations. The key is that "unlimited" rarely means unthrottled; it usually means you get a very high, but still finite, request-per-minute ceiling that resets on a sliding window.
Your exponential backoff with a 1000ms multiplier is likely too aggressive for the initial retry, which compounds your latency issues. Try implementing a tiered approach: first retry at 2 seconds, then 5, then 10, and log the exact timing of each 429. Over a day of logs, you'll likely see the pattern emerge, probably a per-minute limit on the order of 60-120 requests.
For CI/CD workflows, I'd actually recommend moving to an asynchronous pattern. Don't call the API directly in your pipeline. Instead, push tasks to a queue (Redis, SQS) and have a separate worker process consume them with a client-side rate limiter set conservatively below the suspected limit. This decouples your deployment from the external service's throttling.
Have you looked at Anthropic's API? Their rate limits are documented explicitly (requests per minute, tokens per minute), which might offer more predictability for a similar class of model.
SQL is not dead.
The "unlimited" Pro plan definitely has per-minute limits. My logged traffic shows 120 RPM hard cap, no sliding window. If you exceed, you're locked out for the full remainder of that minute.
Your current backoff is part of the problem. Don't retry immediately when you hit 429. You have to wait until the next minute boundary. Add logic to parse the Retry-After header (they sometimes send it) or just sleep for 60 seconds minus the current second.
For batch CI work, pre-calculate your batch size. If you have 500 items, you can't process them in one go. You need to chunk into groups of 120 and send one group per minute. Simple queue.
Otherwise you're just hammering the limit and your p95 will be garbage.
Benchmarks don't lie.