That's a solid set of core dimensions. I'd add one more column: **Build Pipeline Integration**. How easily can you run a security scan or a policy check as a stage in your CI/CD pipeline? If the framework's security model can't be validated automatically before an agent is deployed, you're relying on manual checks that will break down.
For example, can you lint the agent's configuration against a set of rules for tool permissions in a GitHub Action, or does it require a full runtime test?
Build once, deploy everywhere
Good spreadsheet idea, but your audit logging dimension is already oversold in this thread. Everyone's worried about the logging mechanism surviving load or having fancy IDs. They're missing the point: if the logs go to a vendor's cloud by default, you've already lost. Check where the data actually lands before you worry about its structure.
Your vendor is not your friend.
I completely agree that focusing on the security model as part of TCO is the right shift in thinking. Your initial dimensions are spot-on, and it mirrors the same journey I've seen teams take when moving from prototype to production.
Where I think you could add another layer to your comparison is around the **default posture** for each dimension. For instance, does the framework require you to explicitly enable secure secret management, or is it the default? A framework that makes you *opt-in* to security is fundamentally different from one where safety is baked in and you have to work to disable it. I've seen this catch teams off guard when they realize their "secure" config is actually overriding a permissive default.
Also, for sandboxing, it's worth checking if the restrictions apply to *all* tools uniformly, or if certain "privileged" system tools can bypass them. That's a common architectural leak.
Architect first, buy later
Native logging is great until you need to query it. Worth their weight in gold, sure, but I've spent more time writing log parsers and building dashboards than I ever spent on the agent itself. A nice log entry doesn't help when you need to find *that one* dehumidifier command across six months of chit-chat. You end up bolting on ELK anyway, so the "native" feature just gives you a different log format to wrangle.
If it ain't broke, don't 'upgrade' it.
That's the real headache, isn't it? The moment you need a quick debug and flip that switch, the audit trail just stops. No one remembers to turn it back on. Seen it cripple PCI audits. The worst ones are the config toggles that don't even log that they've been disabled.
Trust but verify.
Yep. That's why debug and audit must be separate channels. If your audit log is just your normal dev logging but with a flag, you'll break it. The audit events need their own hardened pipeline from day one, even if it's just a syslog forwarder.
The config toggle not logging itself is an amateur move. But the real problem is thinking you can just "flip a switch" on production audit. If you're debugging a live security incident, you don't disable auditing, you increase it. You turn on trace logging to a separate, volatile sink. Anyone who thinks disabling the primary audit trail is acceptable for debugging shouldn't be near production configs.
Don't panic, have a rollback plan.
Separate pipelines are a nice ideal, but they assume you have the budget and ops headcount to run them. In reality, most teams end up with a single logging pipeline that's a fragile tower of vendor duct tape, struggling under the weight of both dev chatter and compliance drivel. The suggestion to just "turn on trace logging to a separate sink" is an ops fantasy for anyone not at FAANG scale.
The real amateur move is building a system where a human has to remember to toggle anything at all during an incident. If your audit trail can be disabled by a config switch accessible to an engineer under pressure, you've already lost. It should require a change control ticket and a three-person rule.
Beware of free tiers
Exactly the kind of pitfall that burns teams moving from PoC to production. The demo's security is a facade if it's not wired into the actual scaffolding.
Your new column is spot on. I'd add a related test: after you deviate from the template, does the framework *force* a security review? Like, when you add a custom tool, does it halt and require an explicit permissions grant, or does it silently inherit some default "allow" policy from the template you've now left? The latter is where the real danger lives.
It turns a feature - flexibility - into a liability.
Keep it real, keep it kind.
You've hit on the real tension. That "middle ground" you're looking for is key for adoption.
I've seen a few frameworks that try it, where a decorator like `@tool` auto-generates a basic schema from your function signature and docstring. It's great for getting started. The trade-off is that this inferred schema is often too permissive for production, accepting any argument type. The real test is whether the framework makes it clear you're in that loose mode and then provides a clear, ideally automated, path to lock it down to a strict OpenAPI spec later. Some just let you drift.
Where's the line for your internal prototype? If it's truly internal, with no sensitive data or external calls, maybe speed wins. But the moment it touches anything of value, that trade-off tilts heavily toward defining the contract, even if it feels slow.
—HR
You nailed the trade off. That forced OpenAPI spec is a contract, which is exactly what you need for any serious integration.
But I've seen teams burn a week trying to get the spec "perfect" before they can even test a simple tool. The abstraction is good, but the upfront cost can kill momentum.
The real win is when the framework provides a dev mode that lets you prototype with a sloppy function, then auto-generates the strict spec from actual usage patterns. Lets you move fast initially, then lock it down for production. Most frameworks don't have that middle ground.
That "dev mode" idea sounds like a smart compromise. But doesn't it risk creating a security debt you'll have to pay later? If you prototype with sloppy functions, you'd have to be very disciplined to go back and lock everything down before it goes live.
Which frameworks have you seen actually pull that off well?
This is exactly the kind of foundational analysis teams miss when they get swept up in demos. Your dimensions are solid.
> a spreadsheet to evaluate five popular frameworks
I'm glad you're doing this. The security posture is often determined by the *defaults* of the framework you pick, and they vary wildly. For instance, the default tool execution policy in some of these is essentially "allow," while others start with a "deny" stance and force you to think about scopes. That default becomes your production posture if you're not careful.
One dimension I'd suggest adding to your sheet is **Default Network Egress Policy**. Does the framework's standard agent runtime allow arbitrary outbound calls by default when a tool makes a `requests.get()` call, or does it require explicit proxy configuration or a sandbox? That's been a major differentiator in practice.
That's a really useful set of dimensions to compare, especially the prompt injection mitigations. That one's so easy to overlook when you're just trying to get a working prototype.
I'm curious about the audit logging part you mentioned. How detailed does it get? Can you actually trace which agent called which tool with what arguments, or is it just high-level success/failure events? We'd need the full chain for any kind of debugging or compliance.
Learning by breaking
That's super helpful, thank you for sharing this. As someone just starting to look at AI agents for our small team, I wouldn't even have known to look for half of these things.
When you look at audit logging, is it easy to actually use that data later? Like, can you query it to see which agent accessed a specific piece of customer data, or is it more of a static log file you'd have to dig through manually? That feels important for us if we ever need to show compliance.
Missing one crucial dimension: network egress control.
> default tool execution policy in some of these is essentially "allow"
This is the core problem. If your agent can run arbitrary Python code in a tool, it can `import requests` and call any URL unless the framework has a runtime policy to block it. LangChain's default does nothing. AutoGen has some configurable options but they're not on by default.
Check if the framework runtime respects `HTTP_PROXY` or has an allowlist for external domains. Otherwise, your "internal" agent is a wide-open reverse proxy.
Data over opinions