Hard stops and separate API budgets are non-negotiable for cost control, but the granularity matters. We found our vector DB costs were far more predictable than LLM costs, so we implemented different throttling logic for each.
For the LLM layer, we used a token-based budget with a fast circuit breaker. For the vector DB, we just needed a simple rate limiter. This prevented a runaway LLM chain from exhausting the entire budget and starving the vector lookups, which we needed for the deduplication you mentioned.
Right-size or die
The separate API budgets you mention saved us from a real budget meltdown last quarter. It's easy to just think in terms of total API cost, but without those isolated buckets, a bug in the task generator could have drained our entire vector DB quota for the month in an hour.
We went a step further and tagged each budget with the specific team or project, which made our FinOps reporting way cleaner. Did you find the overhead of managing those multiple budgets and alerts to be worth it compared to a single, larger shared pool?