Just built a 'guardrail' agent that watches the main agent. Overhead is 15% but worth it.

I'm a senior data engineer at a mid-sized fintech, and we process millions of transactions daily. Our entire ETL pipeline and analytics layer are driven by orchestrated LLM agents, so guardrails are a production requirement for us, not just an experiment. We run a similar dual-agent setup in production, using GPT-4-Turbo for main tasks and GPT-3.5-Turbo as the critic, but we've evolved it quite a bit from a simple loop.

We went through several iterations of this pattern, so here's a breakdown based on what we learned:

1. **Cost vs. Reliability Trade-off:** Your 15% overhead is actually really good for a first pass. In our environment, the initial naive implementation added about 25-30% more token cost. We got it down to a consistent 10-12% overhead by moving the guardrail's instructions into the system prompt of a smaller, cheaper model and using a structured output (like a JSON schema) for its "pass/fail with reasons" response. This cut down on its verbose reasoning unless a fail state was triggered.

2. **The Hidden Latency Tax:** The 15% more time was the bigger issue for our real-time services. Running agents sequentially introduced too much delay. We moved to an asynchronous pattern where the guardrail agent *validates* while the main agent *generates*, but only for tasks that are purely generative (like writing a summary). For code or SQL, sequential is still safer. For our batch jobs, we just run them sequentially like you do - the extra 10-15 seconds doesn't matter.

3. **Where It Breaks - The Meta Problem:** The guardrail agent is only as good as its checklist. We had an incident where a new type of PII slipped into our data lake, and because "check for PII" wasn't on the guardrail's list, it approved a transformation that exposed it in a new table. You now have to maintain and version *two* sets of instructions: the main agent's and the critic's. We built a small CI pipeline that updates both whenever our data governance rules change.

4. **Integration Effort & The Orchestrator Key:** The biggest leap for us was moving from a Python loop to baking this into our existing workflow orchestrator (Prefect). We created a custom task type that encapsulates the "generate-then-criticize" pattern. This let us reuse it across pipelines, set retries on the critic step separately, and log the critic's feedback for every run. The deployment effort was about 2-3 engineer-weeks to make it robust, but now any team can add a guarded agent task with about five lines of code.

My pick for your use case is to stick with your dual-agent setup, but start wrapping it in a reusable component within your orchestrator. You're getting the core benefit already - that 80% reduction in failures is huge for operational sanity. If your main constraint is keeping costs low, I'd try swapping the guardrail to GPT-3.5-Turbo (if you're on OpenAI) and using a very strict output schema to cut its token usage. If your main constraint is latency for user-facing tasks, tell us more about that - we had to switch to a different, faster pattern for those.

Nice! The 15% overhead for an 80% drop in failures is a fantastic trade-off, especially for batch jobs. I started with a similar loop in my CI/CD pipelines.

One thing that really tightened up our guardrail's performance was adding a structured output constraint to its instructions, like forcing it to return JSON with specific keys (`valid: boolean`, `issues: []`). That cut down on ambiguous "maybe" responses and made the loop logic simpler. We're using Pydantic with the OpenAI client for that.

How are you handling the guardrail's checklist? Is it hardcoded in the prompt, or do you have a way to dynamically adjust the rules based on the task?

Solid start. That 80% failure reduction for 15% overhead is a clear win.

If you haven't, test the guardrail's own failure rate. We had to add a simple rule-based final pass for things like "no absolute file paths" because the critic agent would sometimes miss them. The loop gets stuck if the guardrail itself hallucinates an issue.

How are you defining "failure rate"? Are you measuring silent logic errors, or just runtime crashes?

Great point about the guardrail's own failure rate. We had the exact same issue early on - the critic would hallucinate a security violation, the main agent would "fix" a non-problem, and they'd loop forever. 😅

We measure failure rate two ways: runtime crashes (easy to log) and silent logic errors (harder). For the latter, we sample a small percentage of successful runs for a human review, checking if the output data matches the intended transformation. It's not perfect, but it catches things like off-by-one date filters.

Your rule-based final pass is smart. We ended up doing something similar, but at the *start*: a lightweight regex scan of the generated code for obvious red flags (like 'DROP' or 'rm -rf') before it even goes to the guardrail agent. That saved some pointless critic cycles.

It's a clever hack, but that 15% overhead adds up fast. You're basically paying a 15% "stupidity tax" on every single API call because the primary model can't be trusted to do its basic job.

I always wonder if the real fix is to stop using such an error-prone model for production pipelines in the first place. That 80% drop in failures sounds great until you realize you're starting from a shockingly high failure baseline to need a guardrail in the first place.

Have you benchmarked this against just using a more deterministic, less hallucination-prone model for the main task, even if it's a bit slower? You might find the total cost and latency ends up being a wash.

An 80% reduction in failures for a 15% overhead is a fantastic result - that's the exact kind of measurable trade-off we look for when hardening systems. The "stupidity tax" framing some might use misses the point: even the best models can have unpredictable edge cases, especially when they're dynamically interpreting user requests against a live schema. Adding a deterministic, cheaper check is a classic reliability pattern, like running a linter before a compiler.

Your simple loop is a great foundation. One thing I'd suggest thinking about early is the failure mode when the guardrail *itself* is uncertain. We found that forcing a structured "approve/reject-with-reasons" output, like JSON, helped. But you also need a clear circuit-breaker - if the guardrail starts asking for revisions on a perfectly valid query multiple times, your loop needs to know when to escalate to a log and maybe a human, rather than spinning forever.

What's your plan for when the guardrail and the main agent disagree on a fundamental point, like the very existence of a table? That's where we had to add a fallback to a fast, rule-based schema lookup to break the tie.