Spent the last quarter neck-deep in vendor demos for an AI orchestration layer, specifically looking at platforms that bill themselves as "agent frameworks." The biggest gap I saw between marketing slides and actual, deployable systems was in **agent isolation**. Everyone promises it, but the implementation details—and more importantly, the performance and safety implications—vary wildly.
I stopped taking sales decks at face value and built a weighted evaluation matrix to force concrete answers. The goal was to move beyond "yes we have isolation" to "how, at what cost, and how do you prove it?" This isn't about theoretical features; it's about preventing a runaway agent from spamming your database, burning through API credits, or deadlocking shared resources.
Here's the core of the matrix I used. Each category is scored 0-5, multiplied by the weight. The weights reflect our specific priorities for a high-throughput, multi-tenant internal platform.
### Agent Isolation Evaluation Matrix (Total Weight = 100%)
**1. Resource Isolation & Limits (Weight: 30%)**
* *Sub-criteria:*
* **CPU/Memory Controls (5pts):** Native integration with cgroups/containers (e.g., each agent in its own Firecracker microVM) vs. weaker language-level throttling.
* **I/O & Network Limits (5pts):** Ability to restrict outbound calls per agent, including rate limits on HTTP calls and network egress rules.
* **Cost Enforcement (5pts):** Real-time tracking and hard stops per agent based on LLM token usage or external API call cost. Demo requirement: Show me the dashboard where a kill trigger fired.
* **Sandboxing (5pts):** File system and execution environment isolation. Can Agent A write to Agent B's working directory?
**2. Fault & Error Containment (Weight: 25%)**
* *Sub-criteria:*
* **Process/Sandbox Crash Isolation (5pts):** Does a crashing agent (or its sandbox) bring down the entire worker node, or just that agent's slot? Request logs from a stress test where agents were forced to segfault.
* **Error Propagation Boundaries (5pts):** Unhandled exception in one agent's logic cannot cascade to unrelated agents. Demo ask: Intentionally throw a fatal error in a sub-task—show me the others continue.
* **Deadlock & Timeout Enforcement (5pts):** System-level guardrails that interrupt an agent stuck in a loop, not reliant on the agent's own code to play nice.
**3. Observability & Audit (Weight: 25%)**
* *Sub-criteria:*
* **Per-Agent Telemetry (5pts):** Metrics (CPU, memory, tokens/sec, error rates) exportable to Prometheus with `agent_id` as a first-class label. Not just aggregate platform metrics.
* **Action Logging (5pts):** Immutable, structured log of every external action (API call, DB query) with initiator agent context. Must be exportable.
* **Distributed Trace Integration (5pts):** Agent-level spans integrated into existing OpenTelemetry traces. Can I see the full chain, including the agent's internal reasoning steps?
**4. Data & State Isolation (Weight: 20%)**
* *Sub-criteria:*
* **Session Memory Segregation (5pts):** Clear separation of conversation history, context windows, and working memory between agents operating on the same core. No leakage.
* **Tool Access Scoping (5pts):** Fine-grained, deny-by-default tool permissioning. Agent X can *only* call the Customer DB read API, not the delete API.
* **Vector/Embedding Index Segregation (5pts):** For RAG-heavy agents, mandatory tenant/agent-level partitioning in the vector store to prevent data cross-pollination.
The exercise forced vendors to shift from slides to terminals. The most telling moment was asking them to run a custom load test during the POC: spin up 10 agents, have one go rogue in an infinite loop, and show me the system's response using *their* observability stack. Two platforms couldn't even provide per-agent resource metrics to prove their isolation worked. Another showed a 40% throughput degradation on the healthy agents when the rogue one was terminated—evidence of shared resource contention.
Attached the raw spreadsheet. Copy it, adjust the weights for your context, and force the vendors to prove it. Your ops team will thank you later.
—DL
Benchmarks or bust