Hey folks, been lurking in this subforum for a while. As someone who lives in the observability world, I've seen firsthand how a poorly-chosen ZTNA tool can turn your nice, clean dashboards into a mystery novel. 😅
We just went through a *lengthy* evaluation at my company, and the engineering team decided to open-source the actual checklist we used. It's not a vendor list, but a set of concrete, testable criteria. The goal was to move beyond marketing slides and see how each platform actually behaved. We focused heavily on observability integration, because if you can't *see* what's happening, you're flying blind.
Here's a snippet from the "Logging & Telemetry" section that I, of course, insisted on:
```yaml
requirements:
logging:
connection_events:
must_have_fields: [user_id, device_id, destination_app, decision (allow/deny), policy_id, bytes_up, bytes_down]
must_be_exportable: syslog or s3/gcs bucket or kafka endpoint
agent_telemetry:
must_have: health status, version, last_check_in
must_support: push_to_endpoint (for our existing monitoring system)
observability_integration:
metrics:
required: active_connections, auth_attempts, latency_to_app
export_format: prometheus_exporter or statsd preferred
```
We found that some "big name" platforms treated their own logs as a black box, while others gave us rich, structured data we could pipe directly into Datadog and Grafana. That was a deal-breaker for us. How can you define SLOs for "secure access" if you can't measure error rates or latency per-application?
The full checklist covers stuff like identity provider quirks (Okta vs Azure AD), agent deployment pain, and how "agentless" really works under the hood. Hope it's useful for others in the evaluation phase. Curious what criteria others have prioritized.
Observe all things.
DataDogDodger
Checklists are comforting, I get it. But "must_have_fields" in a YAML spec doesn't guarantee you can actually *use* the data when your pager goes off at 3am.
The last platform we tested spat out JSON with all those fields, but the `policy_id` was a 64-character internal GUID that didn't map to anything in our policy management system. Completely useless for correlation. The logs were "exportable to S3," but on a 15-minute batch cycle, which is a lifetime during a live incident.
You need to test the actual, real-time diagnostic workflow, not just the feature list. Can you, during an active outage, trace a single user's failed connection through every log and metric stream in under two minutes? If not, your checklist failed.
KeepItSimple
Preach. The 15-minute batch export is a classic "checkbox architecture" move. They can technically say it integrates with your data lake, while making it operationally useless.
That policy_id GUID problem isn't an edge case, it's the standard playbook. You need a test in the checklist that forces a live mapping query. Something like: "Trigger a connection, pull the log via real-time API, and use the provided policy_id to successfully query the vendor's admin API for the human-readable policy name within the same script." If you can't automate that validation, the field is just decorative.
Your two-minute trace benchmark is the only one that matters. If the vendor's eyes glaze over when you ask about log stream latency or event ordering guarantees, walk away.
Speed up your build
Nice, a YAML checklist. That'll definitely keep the marketing teams honest.
But a list of fields is just the first trap. The real question is what happens when you query for that `policy_id` across a million log lines at 3am. Does their "export to S3" feature mean you're waiting for a batch job to finish before you can start your analysis? If so, you haven't bought observability, you've bought a post-mortem data dump.
You need a latency requirement. Something like "logs queryable via vendor API within 60 seconds of the event." Otherwise, you're just decorating your data lake.
Keep it simple, stupid