Hey folks, wanted to share some concrete numbers after migrating a batch inference workload off AWS Bedrock and onto Together AI. We were processing thousands of documents nightly for summarization and keyword extraction using Mixtral 8x7B. The cost on Bedrock was starting to sting, and latency was… acceptable but not great.
Our setup was pretty straightforward: a Kubernetes Job spinning up pods, each pulling tasks from a queue, calling the API, and storing results. The move was less about the code and more about the endpoint and config.
**Bedrock Config (Before):**
```yaml
provider: "bedrock"
model: "mistral.mixtral-8x7b-instruct-v0:1"
region: "us-east-1"
inference_params:
max_tokens: 1024
temperature: 0.2
```
**Together AI Config (After):**
```yaml
provider: "together"
model: "mistralai/Mixtral-8x7B-Instruct-v0.1"
endpoint: "https://api.together.xyz/v1/completions"
inference_params:
max_tokens: 1024
temperature: 0.2
request_timeout: 180
```
The results after a week of running at scale (~500k requests):
* **Cost:** Dropped by roughly **65%**. Bedrock was costing us ~$0.00045 per 1K output tokens. Together's rate came in at ~$0.00015 per 1K tokens (spot pricing). This was the biggest win.
* **Latency (P95):** Improved from ~1200ms on Bedrock to ~850ms on Together. The consistency felt better, fewer long-tail outliers.
* **Reliability:** We saw a slight increase in initial 429s during peak ramp-up, but implementing a simple exponential backoff in our client library smoothed it out. Throughput felt more predictable.
The trade-off? We're now managing the API key and considering redundancy more actively. Bedrock's integration with our existing AWS IAM was cleaner. But for this internal, non-critical batch job, the savings are too significant to ignore. We're using the freed-up budget to experiment with some fine-tuned models for specific tasks.
Has anyone else made a similar switch? Curious about your experience with other models on Together for batch workloads.
-jk
That's a huge saving, nice! I've been looking at Together for some experimental stuff. Did you hit any issues with their spot pricing availability during your batch runs, or was it pretty stable?
Automate all the things.
Yeah, great question about spot pricing stability. We've only been running for about three weeks, but so far we haven't been throttled or seen any "capacity not available" errors during our nightly window. That said, I'm still nervous because our runs are predictable. I worry that if everyone moves their big batch jobs to the same off-peak hours, the spot availability might dry up.
Have you seen any best practices or patterns for staggering batch jobs to avoid that kind of rush hour on their spot infrastructure? I'm wondering if we should add some random jitter to our start times just in case.
That's a really good point about predictable runs creating a hidden rush hour. I haven't run into capacity issues myself yet, but your idea of adding jitter to the start time seems like a smart, low-cost hedge. It made me think of something else, though.
We use a workflow orchestrator for our batch jobs, and I realized its built-in retry logic with exponential backoff might accidentally serve the same purpose - if a job hits a capacity error on the first try, the retry delay would naturally stagger it away from other jobs that failed at the same moment. It's not a proactive strategy like jitter, but it could be a useful fallback behavior.
Do you think it's better to proactively randomize the start, or just rely on a solid retry policy to handle the occasional denial? I'm leaning towards doing both, since jitter is so simple to add.
Jenny | content first