We've been running multiple LLM providers (OpenAI, Anthropic, Azure OpenAI, and a smaller local provider) in production for about six months. The use cases are varied: some for internal tooling, others for customer-facing features. I told the team to just build and we'd monitor costs later.
"Later" arrived this week when I finished stitching together our internal dashboard. The monthly run-rate was... a genuine surprise. Not in a good way.
The core issue wasn't the total number, but the distribution. A few patterns jumped out:
* **80/20 rule in action:** 80% of our cost came from just two of our seven deployed models.
* **The long tail of low-volume, high-cost models:** We had a legacy fine-tuned model on a premium tier that was being called a few hundred times a month, but accounted for nearly 15% of the spend.
* **Context window tax:** Using a massive-context model for simple summarization tasks because it was the default in the code.
* **Prompt caching oversight:** We weren't leveraging it for repetitive system prompts, which is basically leaving money on the table.
Here's a simplified version of the query that powers our core view. It's built from our logging pipeline (logs -> BigQuery):
```sql
SELECT
DATE(timestamp) as date,
provider,
model,
SUM(input_tokens) as total_input_tokens,
SUM(output_tokens) as total_output_tokens,
-- Approximate cost calculation (rates stored in a config table)
SUM(input_tokens * input_token_rate + output_tokens * output_token_rate) as estimated_cost
FROM `llm_logs.api_requests`
WHERE DATE(timestamp) >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
GROUP BY date, provider, model
ORDER BY estimated_cost DESC
```
The immediate actions we're taking:
1. **Model Swapping:** Downgrading the legacy fine-tune to a newer, cheaper base model for non-critical tasks.
2. **Standardization:** Enforcing a "model selection" config in our service mesh, pushing teams to use cost-effective defaults unless they get an exception.
3. **Caching Layer:** Implementing a simple prompt cache for high-volume, static-prompt workflows.
Has anyone else done a deep dive like this? I'm particularly interested in strategies for automated model routing based on task type and acceptable latency SLAs. Also, any tools beyond home-built dashboards that have given you clarity?