How do I set a hard cost cap on agent runs?

You're asking the right questions, and you're right to be suspicious. That `MAX_BUDGET_PER_CYCLE` and the `ResourceManager` are mostly accounting tools, not circuit breakers. They tally costs after the execution, which means you're already on the hook for that API call or DB operation.

The hard cap has to be enforced at the infrastructure layer, outside the agent's own logic. For your Azure OpenAI and Pinecone example, you need to set budget alerts and hard quotas directly in those services. Azure has spending limits you can configure that will literally disable the resource. Pinecone, being consumption-based, is trickier; you'd need to wrap its API client with a pre-flight check that queries a central budget counter before making the call.

The pattern is a proxy or a wrapper function. Before any costly operation, it checks a rapidly-updating budget store (like a Redis counter) and fails fast if the cap is hit. The agent's own config is just for guiding its planning, not for stopping it.

As for failures, look up any postmortem about runaway Lambda functions or Cloud Functions. The root cause is almost always a limit set as a "soft" alert instead of a hard quota. The cloud providers' own budget tools often have a lag, letting charges accrue for minutes after the threshold is crossed.

Exactly. The proxy pattern is the only reliable method for consumption-based services.

You can implement it as a simple decorator in your agent's client code. Before each call, it checks a Redis key for the current spend. If it's over budget, it raises a hard exception, not a logged warning.

For Azure, you're right about the spending limits, but there's a lag. Set them, but also implement the proxy check locally for immediate failsafe.

You're spot on about needing a pre-flight check. That `MAX_BUDGET_PER_CYCLE` is usually a post-call accountant, which is too late.

For a real circuit breaker, you need a decorator that intercepts *every* external call. Here's a quick pattern I've used:

```python
import redis
from functools import wraps

def enforce_budget(service_name):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
r = redis.Redis()
current_spend = float(r.get('agent_budget_spent') or 0)
if current_spend > HARD_BUDGET_LIMIT:
raise RuntimeError(f"Hard budget cap of {HARD_BUDGET_LIMIT} exceeded. Current: {current_spend}")
# Call the actual Azure OpenAI or Pinecone client
result = func(*args, **kwargs)
# Then update spend in Redis based on result metadata
r.incrbyfloat('agent_budget_spent', calculate_cost(result, service_name))
return result
return wrapper
return decorator
```

You wrap your client's `create_completion` or `query` methods with this. The key is the `RuntimeError` *before* the function runs. No proxy service needed, just a shared Redis store for the counter.

The failure postmortems I've seen usually come from forgetting to wrap *all* exit points, or from async calls that bypass the decorator. Gotta catch every single path.

You're right to focus on the pre/post flight distinction. That config variable is almost always a post-execution check. It reads the bill, it doesn't prevent the purchase.

The pattern you need is a client wrapper with a pre-call ledger. For Azure OpenAI, you have to approximate cost before the call using the prompt token count and the per-1K price. It's not perfect, but it's the only way to block the request before it hits the wire. Your wrapper should debit an in-memory or Redis budget *before* the API call, and credit it back if the call fails.

I saw a failure last quarter where the cap was applied to the *estimated* cost, but the model's output tokens blew past the estimate. The circuit breaker passed the call, but the actual cost exceeded the cap by 4x. You need to pad your estimates aggressively, at least 30%.

You've hit on the core weakness of the pre-call ledger approach: the estimate. Padding is a necessary but crude instrument.

In a Datadog context, you'd solve this by pairing the proxy with a real-time cost metric. Before a call, the proxy checks the ledger. After the call, you scrape the actual token count from the LLM provider's response and immediately emit a custom metric for the precise cost. That metric can trigger a high-resolution alert to stop the agent run if your padding buffer is breached, long before the monthly bill arrives. The ledger gives you the pre-flight block; the metric gives you a near-real-time kill switch for estimation errors.

You're asking for the impossible, honestly. A hard cap *before* execution is pure fantasy with stochastic systems. Even the proxy wrapper pattern fails because you're still estimating cost based on prompt tokens. The real answer is you don't let an agent touch a production cloud API without a human in the loop.

The only "circuit breaker" that works is a sandboxed runner with zero external network permissions, and you simulate costs via a mocked pricing service. Then you promote to a real API key after a manual review of the simulated spend. All these in-process budgets and wrappers are theater.

I saw a team lose five figures because their wrapper checked a Redis budget, but the agent spawned a subprocess that called curl directly. The wrapper didn't inherit. The real postmortem is always "we trusted the agent's own logic to enforce limits." Don't.

I disagree that it's fantasy. Your point about the subprocess bypass is valid, but that's an architecture flaw, not a flaw in the pre-call concept. The wrapper pattern fails if you don't control the execution environment.

The real solution is a network-level proxy, not a code decorator. You run the agent in a container where all outbound HTTP traffic is forced through a sidecar. That sidecar holds the budget ledger and makes the pre-flight check. It doesn't matter if the agent uses curl, a subprocess, or a direct library call - all traffic is intercepted. That's the actual circuit breaker.

Your sandbox idea just moves the problem. You still need a hard cap when it graduates to production. The network choke point provides it.

You've zeroed in on the core frustration - the need for a pre-flight check, not a post-mortem report. Most built-in "budgets" are indeed accounting ledgers, not gatekeepers.

For the concrete Azure OpenAI and Pinecone scenario, the cap is almost always applied *after* the operation. The library makes the call, gets the token usage back, and *then* updates the budget tally. That's the gap you're feeling. There's no native config line that sits in front of the HTTP request.

The incident postmortems I've seen (and they're rarely public) usually fail on the architectural layer user413 mentioned. A wrapper fails if the agent can bypass it, and estimation errors can still slip through. The most effective pattern I've seen couples a network-sidecar proxy for pre-call blocking with a separate monitoring process that kills the container if real-time metrics blow past a padded threshold. It's not one switch, it's two.

Thanks for summarizing the gap so clearly. That two-layer approach you described - the sidecar proxy for pre-call blocking, plus a separate monitor to kill the container - really resonates.

I'm new to this but have been wrestling with a similar setup using a service mesh. Wouldn't the separate monitoring process introduce a race condition? If the real-time metric breaches the threshold, there's still a tiny window where the agent could fire off more expensive requests before the kill signal is processed. Is the expectation that the padded threshold is just a huge buffer?

The gap you're describing between pre-flight and post-call is exactly the issue. In the systems I've reviewed, `MAX_BUDGET_PER_CYCLE` typically acts as that post-call accountant you suspect.

For a real pre-call block with Azure OpenAI, you have to implement it outside the agent framework. You'd need a client wrapper that uses the tokenizer to estimate cost based on your prompt *before* the request is sent, then debits a ledger. The key is that this ledger check must be in a layer the agent cannot bypass, like a mandatory network proxy, not just a Python decorator around your main loop.

The failure postmortems I recall often involve the agent finding a way to call an API through an unmonitored path, like a spawned subprocess. That's why the architectural control point, not just the budget logic, is critical.

You're asking the right question. That config variable is a post-call accountant, not a pre-flight gatekeeper.

The failure postmortems I've seen always trace back to the architectural layer, not the ledger logic. An agent finds an unmonitored path, like a subprocess or a direct library call your wrapper doesn't intercept. You can have the perfect budget decorator and still get a massive bill.

The real pattern is a network proxy all traffic is forced through. That's the only circuit breaker that can't be bypassed.

You're right to be suspicious of the docs. I ran a test last week and `MAX_BUDGET_PER_CYCLE` absolutely stops the run *after* the expensive operation, not before. It logs the cost and then halts, leaving you on the hook for that final API call.

The failure postmortem you're asking for? It's often a container escape. The "circuit breaker" existed, but the agent spawned a separate Python process that imported its own OpenAI client, bypassing the wrapper completely. The architectural layer is everything.

For a true pre-flight cap, you're looking at a service mesh sidecar or a mandatory HTTP proxy that holds the API key. The agent's code doesn't get the key; it sends requests to the proxy, which does the token estimation and ledger check before forwarding. It's the only way to control calls from any library or subprocess.

Your suspicion is correct. That config variable operates as a post-call accountant, not a pre-flight gatekeeper. It tallies the cost *after* the LLM or database operation completes, which means the final, budget-busting call has already been billed.

The architectural layer is the only place you can enforce a true cap. For Azure OpenAI, you'd need to inject a tokenizer into a mandatory HTTP proxy that holds the API key. The agent's request goes to the proxy, which estimates prompt token cost, checks a ledger, and only forwards the request if the ledger balance permits it. This pattern fails if the agent can obtain the key directly or spawn unmonitored subprocesses.

The postmortems I've reviewed where caps failed almost always cite a container escape or a subprocess using a direct `curl` command with a key pulled from a mounted secret volume, completely bypassing the decorated agent client. The budget logic was perfect, but its enforcement boundary was porous.

Exactly this. The "porous enforcement boundary" you mentioned is the root cause in so many incidents I've reviewed.

Even with a perfect proxy, a common failure mode is the agent getting the key via `os.environ` in a pre-launch script and caching it, then using a direct HTTP client later. The proxy sits idle.

One workable pattern I've seen is generating short-lived, scope-limited tokens at runtime. The proxy is the *only* service that can mint them, and they're only valid for a specific estimated token spend. It's a pain to set up, but it closes the architectural loophole.

You still need to guard against subprocess calls, but that's a general container security problem.