How do I integrate ZPA logs into our existing Splunk SIEM?

Hi everyone, I've been deep in the weeds this past week setting up Zscaler Private Access (ZPA) log ingestion into our Splunk Cloud SIEM. While the official docs point you in the right direction, there are a few configuration nuances and performance considerations I think are worth sharing for anyone on a similar journey.

The goal was to get both Audit Logs (admin actions) and App Connector / Service Edge logs (user connection data) into our Splunk instance for correlation with our other security data. ZPA offers two primary methods: sending logs directly via HTTP Event Collector (HEC) or routing them through an intermediary syslog server. We opted for the direct HEC method to reduce complexity.

Here’s a step-by-step breakdown of the key configuration stages:

**1. Preparing Splunk:**
* First, ensure you have an HTTP Event Collector (HEC) token ready. You'll need to create a new token in Splunk (Settings > Data Inputs > HTTP Event Collector). I dedicated a single token for all ZPA logs.
* The HEC endpoint URL will look like: ` https://your-splunk-server:8088/services/collector/raw`
* A critical step is to **add your ZPA public IPs to the HEC's Allowed List** in Splunk. Otherwise, the connection will be rejected.

**2. Configuring ZPA Admin Portal:**
* Navigate to **Administration > Logs > Forwarding**.
* You'll configure two main forwarders:
* **Audit Log Forwarder:** For administrative events.
* **Connector Log Forwarder:** For app-level session data.
* For each, select **Splunk** as the vendor and **HTTP** as the protocol.
* You'll be prompted for the Server Address (your HEC URL), a Port (usually 8088), and an Authorization Token (your HEC token key).

**3. Log Format & Parsing:**
The logs arrive in JSON format, which Splunk parses nicely. However, you'll likely want to build a custom sourcetype or use `props.conf` to fine-tune the field extraction. Here's a snippet of a basic `props.conf` entry we used for the connector logs:

```
[zpa:connector:json]
TIME_PREFIX = "timestamp":"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
KV_MODE = json
```

**Potential Pitfalls & Tips:**
* **Volume:** App Connector logs can be *very* verbose. Start with a conservative sampling rate in the ZPA forwarder settings if you're concerned about ingest costs or volume.
* **Timestamp Issues:** Pay close attention to time zones in the log payload (`timestamp` field) versus the time they're received by Splunk. We had to adjust the `TIME_FORMAT` to match ZPA's UTC output.
* **Field Extraction:** The built-in JSON parsing is good, but for nested objects (like user information), you might need to write additional `EXTRACT` statements in `props.conf` or use the `spath` command in searches.
* **Testing:** Use the "Test Forwarding" feature in the ZPA portal extensively. It saves a lot of back-and-forth.

After a few days of data flowing, we're now able to correlate ZPA access patterns with endpoint and network events, which has been fantastic for our zero-trust investigations. Has anyone else gone through this integration? I'm particularly curious if anyone has built out elaborate dashboards or alert rules for anomalous private app access they'd be willing to share.