Hey folks, been doing some heavy batch processing lately—think running thousands of documents through an embedding model and then summarizing each with an LLM. My usual combo of OpenAI for summaries and Cohere for embeddings is getting pricey at scale.
I'm evaluating **Together AI** and **Fireworks AI** specifically for high-throughput, cost-sensitive batch jobs. Both seem to offer competitive pricing and claim good throughput. I've run some initial tests, but would love to hear from others in the trenches.
Here's my preliminary setup for a batch summarization task using their Python clients. I'm testing with Llama 3.1 8B variants on both.
```python
# Example batch call pattern I'm using
import together
import fireworks
# Together
together_client = together.Together()
responses = together_client.chat.completions.create(
model="meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
messages=[{"role": "user", "content": batch_prompt}],
temperature=0.1,
max_tokens=256
)
# Fireworks
fireworks_client = fireworks.client.Fireworks(api_key="fw-key")
response = fireworks_client.chat.completions.create(
model="accounts/fireworks/models/llama-v3p1-8b-instruct",
messages=[{"role": "user", "content": batch_prompt}],
max_tokens=256
)
```
**What I'm measuring:**
* **Cost per 1M output tokens** (their pricing pages are clear, but real-world usage can differ)
* **Latency at p95** for batch sizes of 100-1000 requests
* **Reliability** under sustained load over 30 minutes
* **Output quality** for structured extraction tasks (JSON mode is a must)
So far:
* Together's pricing is slightly lower on paper for the Llama models.
* Fireworks seems to have a slight edge in initial latency, but I need to test at scale.
* I've heard Together's batch API endpoint is solid for fire-and-forget jobs.
**Questions for the community:**
* Anyone run large-scale batch inference (>10k requests/job) on either platform? How was the fallback behavior/retry logic?
* Noticed any throttling or degraded performance during peak hours?
* Any gotchas with their JSON mode or function calling for batch processing?
Especially interested if you've compared them on **embedding models** (like BGE) as part of a pipeline. The total workflow cost matters.
I'm Chloem, I run marketing automation for a B2B SaaS company with about 50 employees. We batch-process inbound content daily for lead scoring and summarization, running thousands of documents through embedding and LLM pipelines similar to yours.
- **Throughput and Latency**: Fireworks consistently delivered higher throughput in my load tests. For a batch of 10k summaries, Fireworks averaged about 1500 completions per minute, while Together processed around 900. Latency per request was lower on Fireworks too, about 200ms vs 350ms average.
- **Real Cost for Batch Jobs**: Both advertise competitive per-token pricing, but Together's volume discounts kick in earlier. At my processing volume (about 15M tokens per day), Together came in at $0.12 per 1M tokens, while Fireworks was $0.18. However, Fireworks's higher throughput can reduce compute time costs if you're running on orchestrated infrastructure.
- **Model Availability and Updates**: Together wins on variety, especially for open-source embeddings. I regularly use their BGE embeddings, which are cheaper than Cohere. Fireworks has a narrower but more curated set, with faster deployment of new model versions, often within 48 hours of Hugging Face releases.
- **Operational Reliability**: For sustained, multi-hour batch jobs, I've seen more variability with Together. I've had two incidents in the last quarter where batch jobs slowed significantly, requiring manual requeues. Fireworks has maintained consistent throughput, though their error messaging during rare outages is less descriptive.
I'd recommend Fireworks AI for this specific high-throughput summarization task, as raw speed and consistency matter most for your use case. The cost premium is worth avoiding job requeues. If your batch work heavily uses niche open-source embedding models or you need the absolute lowest per-token cost, go with Together. To make the call clean, tell us your exact daily token volume and whether these jobs are customer-facing or internal backend processing.
Your throughput numbers for Fireworks align with our internal benchmarks, though I'd caution that latency per request becomes less meaningful at scale than tail latency and consistency. We've seen Together's p99 latency spike unpredictably during sustained batch loads, while Fireworks maintains tighter variance.
On cost, that 50% price difference at your volume is significant, but the throughput advantage can shift the equation. If your orchestration is cost-sensitive to wall-clock time - think hourly cloud function execution or managed Airflow slots - Fireworks' speed might close the gap. Have you quantified your infrastructure cost per compute hour? We found that for our BigQuery-based pipelines, faster processing reduced slot consumption enough to offset the token premium at around 20M tokens daily.
The model update speed you mentioned for Fireworks is underrated. Their rapid deployment of new versions - we got LLaMA 3.1 70B just 12 hours after Meta's release - let us patch critical bugs mid-batch without retooling our pipeline. Together's wider selection is great for experimentation, but production reliability favors Fireworks' curated approach.
data is the product