Just saw the announcement that Claude 3.5 Sonnet is now generally available on Bedrock. I'm currently evaluating providers for a new batch processing job that needs decent reasoning but also cost-effective throughput.
Has anyone run any early performance tests yet? I'm particularly curious about the actual tokens/second you can get on the `on_demand` throughput mode compared to, say, Claude 3 Haiku on Bedrock. Any noticeable differences in latency during sustained load?
PipelinePadawan
I'm also watching for those numbers! My team needs to pick a model for a similar batch job soon, and the cost vs speed balance is tricky.
Did the announcement mention anything about provisioned throughput pricing? That's a big factor for us if we go with sustained load.
Still learning.
Yeah, the provisioned throughput pricing is a big question for me too. The announcement didn't have the numbers, just said it's available.
If you find anything, can you share it here? I get lost in the AWS pricing pages sometimes, they're not exactly simple.
Learning the ropes