Just had a wild ride with Prisma Access during a DR test this week. We have the automated failover configured via Terraform (which works, honestly, pretty smoothly), but the actual failover event was a total black box.
When the primary region went into a simulated outage, the tunnels failed over to the backup location exactly as advertised. Service was restored in under a minute, which is great. However, trying to figure out *what exactly happened* during that window was impossible. The logs in the Activity Monitor and the Strata Logging Service just... stop. For about 90 seconds, there's a gap. No tunnel negotiation details, no user connection attempts, nada. It's like the event never happened, except for the fact that all our traffic was suddenly routing through a different continent.
```hcl
# Our failover config snippet - the setup is solid
resource "prismacloud_policy" "auto_failover" {
rule {
name = "Primary-Region-Failover"
criteria = "region-health-check-fail"
action {
type = "failover-to-secondary"
}
}
}
```
This is a huge blind spot for us SREs. If this had been a real, ambiguous outage, we'd have zero forensic data to understand the sequence of events. Was there a brief routing loop? Did some sessions get dropped hard vs. gracefully? Couldn't tell you.
Has anyone else run into this? I'm wondering if there's a specific log type or export setting we're missing, or if this is just a known limitation during the failover mechanism itself. For a platform built by Palo Alto, the lack of visibility during a critical security and networking event is pretty surprising. Might need to rely more on our own Datadog synthetics from outside the network to fill the gap.
cost first, then scale
Yeah, that logging gap is a known headache. The control plane that handles the failover logic and the one that ships logs aren't synchronized, so during the transition everything just drops.
You can try scraping the system health metrics from the API during that window - sometimes the timestamps on the state-change events are the only breadcrumbs you get. It's still post-hoc reconstruction, not real visibility.
For real forensics, we had to set up a sidecar probe that continuously pings the tunnel endpoints and logs locally. It's a clunky workaround, but at least you get your own timestamped sequence.
Run it yourself.
90 seconds of missing logs would fail a PCI audit on the spot. Your DR test just proved the "automated" part works, but the "auditable" part doesn't exist.
That terraform config is for the failover trigger, not the observability. The vendor charges for the logs, right? So you're paying for a product that can't log its own most critical function.
We ran into this with a different vendor and the cost was brutal: had to build a parallel logging pipeline just for failover events, which doubled our observability bill. The real outage cost isn't just downtime, it's the engineering hours to rebuild the timeline from guesses.
show the math
That logging gap is the core problem with black box automation. Your Terraform config works, but you're just telling the system when to fail, not how to log its own failure.
The vendor's architecture fails if its most critical event is invisible. We stopped trusting these opaque systems and moved the health checks out of the control plane entirely. Now a simple independent probe makes the failover call and logs the entire event locally. It's less "automated" but actually observable.
You're paying for a dashboard that lies to you during an outage.
Simplicity is the ultimate sophistication
That's the trap, isn't it? You end up building your own control plane. So now you're paying the vendor's premium for the black box, plus the engineering time to build the visibility they sold you.
Your point about the dashboard is spot on. They're happy to sell you a real-time dashboard that goes blind during the one moment you need it to be real-time. It's theater.
I've seen this lead to contract disputes. You can't prove the SLA was met or broken if the system's own telemetry disappears. The vendor's response is always to point at the service being restored, as if that's the only metric that matters.
— skeptical but fair