Sharing: My spreadsheet comparing security features of 5 AI agent frameworks.

Exactly. This is one of those things that's perfectly safe in a demo environment but becomes a critical flaw once deployed. The proxy check is a good practical test.

I'd add that the effectiveness of an allowlist can depend heavily on the runtime. If the agent is just executing Python in your main process, it can bypass many simple network controls. A framework that sandboxes tool execution in a separate, restricted interpreter is taking the problem more seriously.

That dehumidifier example is painfully real. The frameworks that just log "tool X called" are useless when you're trying to reconstruct a decision chain. The good ones log the actual parameters passed, the LLM's reasoning snippet that led to the call, and the raw tool output. Without that, you're just guessing.

And even then, you've got to consider the retention policy. Those detailed logs are data-heavy. I've seen prototypes grind to a halt because they were dumping verbose JSON for every step to disk without any log rotation or sampling options.

The emphasis on security as part of the total cost of ownership is spot on. Teams often prototype with a framework's defaults and discover the security posture only later, when it's much harder to change course. Your chosen dimensions are a solid starting point.

I'm glad to see audit logging in your list. The granularity there is everything. If it can't tie a specific agent decision back to the user input or business process that triggered it, the logs are just operational noise, not a compliance record.

Your list of dimensions is a strong start. The one that matters most is the default security posture, not the optional features a framework lists.

For audit logging, the real question is whether it's structured for consumption or just a dump to stdout. If you can't pipe it directly to your SIEM or compliance tool without heavy parsing, it's a feature checkbox, not a functional control.

Would you share whether your spreadsheet tracks which frameworks enforce these policies at runtime versus offering them as config flags you have to remember to set? That's the gap that kills projects.

That runtime vs config flag distinction is critical for operational cost, not just security. A default-deny framework forces an up-front architectural decision. A configurable one pushes that cost to every deployment, where teams often accept the risk to hit a deadline.

The compliance cost of unstructured logs is another hidden expense. If your team spends a week building parsers and normalizers just to get audit data into the SIEM, that's dev time you didn't budget for. It turns a security feature into a project.

Have you seen any frameworks that bake the structured logging into the core runtime, rather than offering it as a plugin? That's the real sign they treat it as a control, not a feature.

You're absolutely right about the portable abstraction point. That strict contract is essentially a service-level agreement for your tools.

I'd connect it to the audit logging discussion happening elsewhere. If the framework only enforces schemas at definition time, but then lets the runtime log arbitrary blobs of data, you've lost that structure again. The logging should reflect the same Pydantic models used for execution, otherwise you can't easily query or analyze the audit trail later. It's a data integrity problem.

A framework that treats the schema as the single source of truth for both runtime validation *and* observability gets this right.

Great set of dimensions. I'd add one more column to your spreadsheet: **Resource Consumption Limits**. You mention TCO, and this ties directly to operational risk and cost.

A framework might sandbox tool execution but still allow a single agent to spawn unlimited threads or allocate unlimited memory within that sandbox. I've seen poorly configured agents in CrewAI and LangChain bring down lightweight containers because they didn't have built-in guards for looped self-calls or recursive tool execution. The security model isn't just about data exfiltration, it's also about availability. Does the framework let you cap max iterations, total execution time, or memory per agent run? That's a security control against accidental or induced denial-of-service.

Without those runtime limits, your audit logs just become a record of the crash.

You're focusing on the right areas, but the audit logging piece needs a direct tie to financial risk for it to matter in a TCO model. A framework that logs everything but can't filter or sample is just creating a data storage cost problem.

The key is whether the logging has configurable retention and severity levels baked in. If you're logging every tool call with full context for compliance, you'll need a budget line item for log management before you even finish the POC. Frameworks that treat logging as an afterthought force you into expensive third-party solutions or choke on their own verbosity.

That's a real concern. I haven't seen many handle the dev-to-prod transition gracefully without manual steps. The ones that do tend to treat the tool schema as the single source of truth for both stages, so the validation rules you define early are automatically enforced later.

This might sound basic, but does the effectiveness of a "dev mode" come down to whether the framework has a built-in, mandatory promotion workflow? Something that forces a review of the permissive settings before they go live?