In the course of our annual security audit cycle, we have been evaluating various model providers for a new internal tool that will automate portions of our compliance evidence collection. This tool is expected to generate significant, unpredictable bursts of API traffic—potentially exceeding one thousand concurrent requests—as it parses logs, summarizes control gaps, and drafts remediation tickets across multiple business units simultaneously. Given the criticality of this function to our audit timeline, provider reliability under sustained and peak load is a non-negotiable requirement, ranking equally with output quality for our specific domain.
We have shortlisted Together AI based on preliminary cost-per-token analysis and their model offerings, which appear suitable for our mixed workload of classification and summarization. However, our vendor security review process mandates a thorough operational resilience assessment before proceeding to contract. While latency percentiles and throughput are commonly published, real-world behavior during failure modes and recovery is often undocumented.
I am seeking empirical, practitioner experiences regarding Together AI's infrastructure, specifically:
* **Error rate progression** as concurrent calls scale from hundreds into the thousand+ range. Does the service degrade gracefully or exhibit a sharp cliff-edge failure?
* **Retry behavior and backoff recommendations.** Are `429` or `5xx` responses accompanied by useful `Retry-After` headers? What has been your effective strategy for implementing exponential backoff with jitter in this environment?
* **Consistency of latency** at the 95th and 99th percentiles during sustained load over periods of 30+ minutes. We are less concerned with median latency and more focused on tail latency predictability.
* **Dependency transparency.** During any observed outages or degradations, was the root cause communicated with sufficient technical detail (e.g., specific cluster, hardware failure, software deployment) to inform our own risk assessments and incident reports?
* **Geographic redundancy experience.** For those utilizing multiple regions or endpoints, how effective is failover, and is there observable data locality performance loss?
Our internal checklist for this evaluation phase includes seventeen distinct reliability criteria, but the above points represent the core of our operational concerns. We are particularly interested in scenarios that mirror compliance-related workloads, which are often non-uniform and demand high consistency rather than raw speed. Any data or anecdotes regarding sustained throughput over several hours, the behavior of streaming responses under load, or the effectiveness of their load balancers would be immensely valuable.
—at
—at
That's a critical requirement for your use case. While I haven't pushed to exactly 1k concurrent calls, I've observed their system handle bursts in the low hundreds for a batch processing job. The main bottleneck wasn't the API gateway, but rather throttling at the model level depending on which specific model you're calling. Their newer, popular models can hit quota limits faster than you'd expect during sustained peaks.
You mentioned recovery behavior, and that's a good point. In my experience, their retry logic and error messaging during overloads were standard - HTTP 429s with a `Retry-After` header. There was no graceful degradation, just a hard queue. For your timeline-critical function, you'd need a very robust queuing and retry layer on your client side, treating it as an eventual-success system during their peak windows.
Have you looked at their enterprise SLA terms? They differ significantly from the pay-as-you-go tier and might give you the commitment on capacity reservation you need.
Keep it real
Your focus on undocumented failure behavior is exactly where the real pain lives. I've seen their newer models like Llama 3 70B get swamped at well under your target concurrency, turning those "unpredictable bursts" into predictable cascading failures if you're not careful. Their rate limits aren't just per-account, they're per-model, and the documentation is often playing catch-up with the actual queue depths.
You absolutely cannot treat this as a simple API call. You need a client-side queue with exponential backoff, model fallbacks (if one is slammed, try another in your plan), and aggressive telemetry on 429s versus 5xx errors. For a timeline-critical audit function, you're essentially building a mini load balancer and circuit breaker in front of their API. Their SLA doesn't account for your internal timeline when a model is saturated.
Frankly, if you're mandated to prove operational resilience, you should budget for a two-week load test simulating your worst-case burst patterns before signing anything. Their dashboard metrics are smoothed averages, not p99 latency during a stampede.
Speed up your build