We're looking at a major workflow automation project that will use LLM calls for data enrichment and classification. I need to decide between self-hosting an open-weight model via an API gateway or just paying for a managed API.
I've ruled out the big proprietary providers (GPT-4, Claude) for this due to cost at scale and data privacy requirements. The open-weight landscape is a mess of benchmarks that don't reflect real-world use.
My core question: which open-weight model family currently offers the best balance for a high-volume, low-latency production API?
I need specifics, not hype. If you've run something like Llama 3.1, Qwen 2.5, or Gemma 2 in production, I want your actual numbers. My criteria:
* **API Quality:** The ecosystem around the model for a *clean* OpenAI-compatible API endpoint. I don't want to hack together a Frankenstein setup.
* **Inference Performance:** Tokens/sec on standard cloud GPU instances (e.g., A10G, L4). Quantized vs. full weights.
* **Operational Overhead:** What does it take to keep it running? Update nightmares, memory leaks, etc.
* **Managed Alternative:** If a managed API exists for this model (e.g., Together, Anyscale), how does its cost compare to the self-hosted total cost of ownership?
For example, a baseline I'm considering:
```yaml
Model: Llama 3.1 8B Instruct (Q4_K_M)
Hardware: 1x L4 GPU (24GB)
Tool: vLLM with OpenAI-compatible server
Observed: ~120 tokens/sec, ~2GB VRAM for context
```
Is this representative? Is Qwen 2.5 7B consistently faster? Does Gemma 2 9B have a more stable serving stack?
I'm particularly skeptical of "70B" parameter claims that ignore the hardware required for feasible throughput. If your "best" model requires 4x H100s, that's not an answer for a cost-per-token discussion.
What's running your production workloads right now, and what's the actual bill?
Show me the query.
I'm a marketing ops lead at a mid-sized e-commerce company (around 120 people), and for the last six months I've been running a self-hosted Llama 3.1 70B model via vLLM for product description tagging and support ticket categorization. We also tested Qwen 2.5 72B and used Anyscale's managed Llama API for a proof-of-concept.
**API Quality:** For a clean setup, Llama 3.1 with vLLM is the easiest path. The OpenAI-compatible API server launches with one command, and it's genuinely plug-and-play for swapping from OpenAI's endpoints. We had more configuration headaches with Qwen's TensorRT-LLM setup, needing extra tuning for the chat template.
**Inference Performance:** On AWS g5.12xlarge (4x A10G), our 4-bit quantized Llama 3.1 70B serves at ~85 tokens/sec for typical 512-token inputs. The same instance with Qwen 2.5 72B was slightly slower, around 65-70 tokens/sec. For full precision, expect a 35-40% drop.
**Operational Overhead:** This is the hidden cost. vLLM is stable, but updating the model weight files (like moving from 3.0 to 3.1) requires a full service restart and careful version-checking of libraries. We saw a 2-3% memory creep over a week, solved by a scheduled weekly restart. You'll need a dedicated devops cycle for this.
**Managed Alternative:** Anyscale's managed Llama 3.1 70B costs about $0.80 per 1M input tokens. For our volume, self-hosting on that g5 instance was about 30% cheaper, but that doesn't factor in my team's engineering time for maintenance, monitoring, and upgrades. The managed API had near-zero latency variance, while our self-hosted endpoint sees occasional 200-300ms spikes during garbage collection.
My pick is to start with a managed API (like Anyscale or Together) for Llama 3.1 for your first major project, unless you have a dedicated SRE person. The operational overhead is real and will distract from your core workflow automation build. If you have a strict per-token budget under $0.50/M and guaranteed in-house ops bandwidth, then self-host with vLLM. To decide cleanly, tell us your expected daily token volume and whether you have a person who can own the model infrastructure full-time.