So the security team bought Perimeter 81 and now wants every network event funneled into our SIEM. Another "single pane of glass" to babysit. Fine. Here's how we got the logs from P81 into Splunk, since their docs are all marketing fluff.
P81 has an API for audit logs. You'll need a service account token from their admin panel. We pull logs hourly via a simple Python script in Airflow, because their "webhook" push was flaky. Script hits the `/v1/audit/logs` endpoint, paginates, dumps raw JSON to S3.
From there, it's a standard ETL: S3 -> Staging Table (BigQuery) -> transformed view -> Splunk HEC. The raw schema is a mess of nested objects. Flatten it with SQL, not in your application code.
```sql
-- Example flattening for user connection events
CREATE VIEW p81_connections AS
SELECT
timestamp,
JSON_VALUE(event_data, '$.origin.ip') as source_ip,
JSON_VALUE(event_data, '$.resource.name') as gateway_name,
action,
actor.email
FROM `raw.p81_audit_logs`
WHERE category = 'CONNECTION';
```
The main pitfall? Their timestamp field is UTC but comes as a string with microseconds. Cast it properly or your Splunk alerts will be useless. Also, the API rate limits are low—so much for real-time alerting. Now we get our "critical" alerts roughly 15 minutes late. Impressive.
SQL is enough
That timestamp issue is a classic gotcha. Had the same problem with their admin audit logs. The field is `created_at` for those, and it's a string format that BigQuery's PARSE_TIMESTAMP handles if you specify `%Y-%m-%dT%H:%M:%E6SZ`.
You mentioned the low API rate limits - we hit that hard during initial backfills. Had to implement exponential backoff with jitter in the script, otherwise you get throttled after about 15 minutes of pulling. Their support won't increase it for you either, it's a platform-wide limit.
One more thing: watch the `action` field mapping. It's not consistent across event categories. A "CREATE" for a user is different from a "CREATE" for a gateway. You'll need a lookup table or you'll mislabel alerts.
Excellent point about the `action` field inconsistency. We built a dedicated mapping table for that exact reason, and it still requires quarterly updates as they add new event types. The more subtle issue we found is with the `user_id` field across different log streams. In connection logs, it's the internal P81 UUID, but in some admin audit entries, it can be an email address or even null for system actions. This breaks any downstream correlation unless you normalize it early.
Your note on the platform-wide rate limit is spot on. We've had to schedule our extraction jobs not just with backoff, but also to run during off-peak hours for their region, which we inferred from increased latency. Even with that, for large tenants, a full historical backfill may require negotiating a special export directly from their support, though they frame it as a one-time "compliance request."
Have you also observed discrepancies in the geo-location data between the connection logs and the gateway status API? We've had cases where the IP city/country in the log doesn't match the gateway's reported location, creating confusion for our geo-fencing alerts.
—at