I've been running some systematic batch processing workloads to compare LLM API providers, focusing squarely on the mid-market use case: processing 10k-100k documents per job, with average prompt+completion tokens around 2k per document. The goal is reliable, non-realtime extraction and summarization.
My test harness is built on Apache Spark, distributing API calls across a cluster, with careful rate limiting and retry logic. I measured total cost and effective throughput (documents processed per dollar) across several major providers for their most cost-effective batch-oriented models. Here are the key findings from my last run:
* **Claude 3 Haiku (Anthropic)** delivered the best raw cost-per-token for mixed prompt/completion workloads in my test. The simplicity of their per-token pricing made projections straightforward.
* **GPT-3.5-Turbo-Instruct (OpenAI)** was very competitive on pure extraction tasks, but its lack of a system prompt can complicate some batch patterns.
* **Gemini 1.5 Flash (Google)** showed strong numbers, especially when using their native batch API feature, which reduced overhead costs significantly.
The critical factor was tuning the batch size and concurrency in my Spark jobs to match each provider's rate limits without incurring penalties. For example:
```python
# Spark UDF structure for batch calls
def process_batch(document_batch, model_endpoint, api_key):
headers = {"Authorization": f"Bearer {api_key}"}
payload = {
"model": "claude-3-haiku-20240307",
"messages": [{"role": "user", "content": doc} for doc in document_batch],
"max_tokens": 500
}
# Batch size optimized per-provider
response = requests.post(model_endpoint, json=payload, headers=headers)
return parse_responses(response)
```
Has anyone else done similar large-scale batch cost comparisons? I'm particularly interested if others have factored in the cost of failed requests and retries, or have data on providers like Cohere or Mistral's API for these workloads. Sharing your architecture and throughput numbers would be valuable.