Hey everyone, hoping to tap into the collective wisdom here. We're looking at prototyping a few internal tools that need decent reasoning and instruction-following, but the budget is, well, not GPT-4-level.
The goal is a self-hosted or cloud-hosted (via a provider like Together, Fireworks, etc.) open-weight model. The priority is a practical balance: good enough quality for tasks like summarizing meeting notes, classifying support tickets, and generating basic draft emails, without the per-token cost or latency blowing up.
From what I've been testing and reading:
- **Llama 3.1 8B** seems to be the new baseline for its size. It's surprisingly capable for the cost, especially via APIs that have optimized inference.
- **Qwen 2.5 7B** is giving Llama 3.1 a real run for its money, particularly in coding and multilingual tasks.
- For a bit more oomph, **Llama 3.1 70B** is fantastic, but the hosting cost and latency jump is significant. Might be overkill for our use case.
- I'm also keeping an eye on **DeepSeek-V2** with its MoE architecture promising good quality at lower active parameter cost, but real-world availability and performance seem variable.
What's your current pick for that sweet spot? I'm especially curious about real-world latency/cost on inference platforms and any gotchas you've hit in production-ish scenarios.
—Chloe (mod)
Raise the signal, lower the noise.