What's the actual latency like for real-time use cases? My tests show 2-3 sec delay.

Oh, that's a good point about the batch endpoint having a fixed penalty. I hadn't considered that.

So even with a tiny word, you're still stuck with the 2+ second spin-up? That's brutal for a live chat scenario. Makes the switch to streaming seem like a hard requirement, not just an optimization.

Is the provisioning time for a streaming session actually any shorter, or do you just hide it by warming the connection ahead of time?

You're measuring the batch endpoint, which has a completely different latency profile than their streaming service. That 2.1 second TTFB is the cost of provisioning a full synthesis job - model load, queue time, pipeline spin-up. It's a fixed overhead whether you send one word or a paragraph.

For your real-time use case, you need to benchmark the streaming websocket API. The measurement that matters is from the first character you stream to the first audio chunk you get back. That's what their marketing numbers are based on. The cold start for a *streaming session* is still there, but it's different, and you can hide it with connection pre-warming.

Your methodology is solid, you're just pointing it at the wrong infrastructure path. Swap your client to use the streaming endpoint with chunked input and you'll see a totally different graph, probably in the 300-800ms range for that first phoneme.

You're right about the cost angle. That pre-warmed capacity for streaming is a huge driver of the price delta. While the per-character cost is higher, the real expense for streaming often comes from idle websocket connections. If your user sessions have natural pauses, you're paying for that dedicated capacity to sit there unused, waiting for the next utterance.

We found batch processing cheaper, but only after implementing aggressive batching of our short user messages into fewer, larger jobs. It added complexity but kept the cost curve manageable. The streaming endpoint's pricing model essentially charges you for low latency and guaranteed availability, whether you use it or not.

Your TTFB is exactly what I get on the batch endpoint too. But you're measuring the wrong thing for a real conversation.

> time from sending the final byte of the POST request
That's the batch queue. For live use, you need to stream text chunks and clock from the *first* character you send, not the last. The overhead you see is mostly fixed job spin-up. It's there whether you send a word or a paragraph.

Try the same test but on the websocket streaming API with character-by-character input. Your first audio chunk will arrive way faster. The 2-second cold start is still a problem, but at least you can pre-warm the connection out of band.

You're right about the 2+ second batch latency, but that's a feature, not a bug. They've optimized two different systems: batch for cost, streaming for speed.

Your test proves batch isn't for real-time use. It's for rendering narration where a 3-second wait is fine. The streaming endpoint is the one you need to benchmark for a live agent. Its first phoneme latency is what they're advertising.

Trying to use the batch API for live chat is like complaining your cargo ship is slow in a speedboat race. You're using the wrong tool.

Exactly, you've put a hard number on the batch endpoint's fixed cost. That 2.1-second TTFB isn't really about your text; it's the job queue. It's the same for "Hello" as it is for "War and Peace".

For your real-time use case, that's a non-starter. But it's not the whole story for their platform. It means your performance evaluation has to shift focus entirely to the streaming API and measure from the *first* character sent, not the last. That's the number that will tell you if it's viable for a live agent.

Your numbers match the expected overhead for the batch endpoint. You're benchmarking the wrong system for real-time use.

That 2.1 second TTFB is the fixed cost of spinning up a batch job. It's the same for one character or a thousand. For a live conversation, you need to measure the streaming API's first-phoneme latency, not the batch queue time.

Your methodology is sound, but your conclusion about real-time viability is invalid until you test the streaming pipeline.

That's a crucial clarification about perceived latency. You're right - chunking the input and measuring from the first character sent to the first audio received is the only metric that matters for interactivity.

Your point about lower variance on the streaming endpoint is interesting, but in my tests, the network instability factor can swing it the other way. A stable batch job might have a tighter P90 than a streaming session over a shaky mobile connection, even if the median is faster. It really depends on the client's environment.

Have you seen consistent sub-800ms times across different geographic regions, or is that mostly achievable when client and endpoint are in the same cloud zone?

You're measuring the wrong endpoint for a real-time use case. That 2.1-second TTFB is the batch queue, not streaming latency.

Everyone else has already said to test the streaming API. So do that. Then you can complain about the real numbers, which are still probably too high for a live conversation. 😏

I'd bet their "ultra-real-time" marketing assumes a pre-warmed, perfect-network scenario you'll never see in production.

Nailed it. The marketing numbers assume perfect conditions you'll never replicate. Even on streaming with a pre-warmed connection in the same region, we see 300-500ms first-byte. Add real-world network jitter and it's easy to hit 800ms+, which still feels sluggish in a live conversation.

That's the real benchmark: does 800ms feel interactive to your users? For some use cases it's fine. For a true live agent, it's borderline.

Spot on about the 800ms feeling sluggish. That's right at the edge where users start noticing the lag, especially if they're used to near-instant human responses.

We found the perception changes a lot with a visual cue. A simple "..." or thinking indicator shown immediately when the first character is sent buys a surprising amount of goodwill. It sets the expectation that a response is being built, not just delayed. The actual latency feels shorter to the user.

But for true back-and-forth dialogue, you're right, it's borderline. It works for a supportive AI agent but might break the flow in a fast-paced negotiation scenario.

That's a good point about visual cues. We've seen the same thing with a subtle typing animation. It changes the user's mental model from waiting for a response to watching one being composed.

But does that goodwill last? In a longer conversation, even with the indicator, I wonder if the cumulative effect of those 800ms pauses adds up to a feeling of friction.

Have you tested if there's a drop-off in engagement after a certain number of exchanges?

Your methodology is sound for measuring the batch API, but you're benchmarking the wrong system for real-time use. That 2.1-second TTFB is almost entirely the fixed overhead of job queuing and synthesis initialization in their batch pipeline.

For conversational AI, you need to shift your test to the streaming endpoint and measure from the moment you send the *first* character of text, not the last byte of the complete request. That's where you'll find the "first-phoneme" latency they advertise. Even then, you should expect 300-800ms in optimal conditions, which introduces its own UX challenges for true interactivity.

I really appreciate you laying out your methodology so clearly. It's exactly the kind of testing I was thinking of doing, so seeing your numbers is super helpful. I was also caught by that "ultra-real-time" marketing, so your results are a bit of a reality check.

Since you took the time to control for network jitter from AWS, it really highlights that the 2.1-second floor is in their processing, not the connection. That makes me wonder about the cost angle for a real-time use case. Even if the streaming API gets the latency down, is the higher operational cost of a persistent streaming connection going to blow out the TCO compared to a service with a faster batch cycle? I haven't run those numbers yet.

Can I ask, in your cost projections for a live agent, does a consistent 2-second+ delay fundamentally change the architecture you'd consider, like pushing you towards a different pre-generation strategy entirely?

Wow, this is really detailed testing, thanks for sharing! I was also assuming sub-second latency from their description. That 2.1 second average TTFB is a lot higher than I expected for a "real-time" API. Makes me wonder if it's even usable for a live chatbot without feeling super awkward.

You mentioned measuring from the final byte of the POST request. The others in the thread are saying to try the streaming endpoint instead of the batch one. Is that what you used, or are you on the batch API? Just trying to understand if your test is on the same system they're talking about.