Hey folks, been deep in the weeds this past month setting up Prisma Cloud's runtime defense alerts to actually *do* something useful beyond email noise. Our team lives in VSCode and Slack, but for critical infra alerts, the escalation path is dead-serious PagerDuty. I wanted to share the step-by-step we landed on, because the documentation felt a bit... scattered across both Prisma and PagerDuty's UIs.
The goal was straightforward: get a high-severity runtime alert (think a critical vulnerability exploitation attempt or a malicious process spawn) to trigger a PagerDuty incident, which then pages the on-call engineer based on our schedule. The tricky part was filtering the signal from the noise and formatting the payload so PagerDuty could understand it.
Here's the core workflow we built:
1. **Prisma Cloud Alert Rule:** We created a dedicated rule for runtime defense, scoped to our production accounts. The key was using the RQL filter to be super specific.
```sql
alert rule create "Runtime Critical - PagerDuty"
--description "Triggers on CRITICAL runtime events for production"
--severity "critical"
--event-type "Runtime"
--filters '{"timeRange":{"type":"relative","value":30},"query":"config from cloud.resource where cloud.type = 'aws' AND api.name = 'aws-cloudtrail' AND json.rule = 'Runtime Alert' AND threat.severity = 'CRITICAL'"}'
--notification-channel "our-pagerduty-integration-id"
```
2. **PagerDuty Integration Setup:** In Prisma Cloud, you add a "PagerDuty" notification channel. This requires the PagerDuty Integration Key from a *Service* you create in PagerDuty. We made a new service called "Prisma Cloud Runtime Defense." The payload mapping is crucial—Prisma sends a giant JSON blob, but PagerDuty only needs specific fields.
3. **Payload Customization (The Fun Part):** The default payload mapping was lacking context. We used PagerDuty's "Custom Incident Action" feature to transform Prisma's alert. Here's a snippet of the `custom_details` we extracted to make the incident actionable:
```json
{
"alert_id": "{{.ID}}",
"cloud_account": "{{.CloudAccount}}",
"resource_name": "{{.ResourceName}}",
"threat_severity": "{{.ThreatSeverity}}",
"threat_type": "{{.ThreatType}}",
"rule_name": "{{.RuleName}}",
"prisma_cloud_link": "{{.AlertLink}}"
}
```
This gives the on-call engineer everything they need to click straight into Prisma Cloud for investigation.
**Pitfalls we hit:**
* The Prisma-to-PagerDuty integration uses PagerDuty's v1 Events API, which is fine, but the error logging if your payload mapping fails is... minimal. Test with a mock alert first.
* Initially, we got flooded because we didn't scope the RQL tightly enough. Remember, runtime events are *noisy*. Start with just `CRITICAL` severity.
* The alert link in the payload is golden, but ensure your engineers have the right Prisma Cloud permissions to access it, or they'll hit a login wall at 3 AM.
The integration has been solid for about three weeks now. We've had two legitimate pages that caught real, nasty behavior in our containers. The rest is filtered out. It feels like we've finally plugged a major visibility gap in our runtime security.
Has anyone else set up something similar? I'm curious if you used the Prisma Cloud API directly for more flexibility, or perhaps routed it through a middleware like AWS EventBridge first for additional transformation.
editor is my home
Filtering for the signal is the right move, but I hope you're also tagging those runtime workloads by cost center. I've seen teams page the on-call for a critical alert, only to spend an hour discovering the impacted resource runs on a dev instance that costs $4 a month. The response effort shouldn't cost more than the asset.
Did you set up a severity downgrade for alerts tied to low-cost or untagged resources? That's our next step - stops us from getting woken up for a critical on a sandbox.
Cloud costs are not destiny.
Filtering for the right signal is a huge first step, and I completely agree the tagging piece is what makes it sustainable. We actually built our severity logic directly into the Prisma alert rule using tags, so it creates the PagerDuty incident with the correct priority from the start.
For us, any resource tagged `environment: sandbox` or `cost-center: unknown` automatically downgrades the incident to a PagerDuty "low" priority. That keeps it out of the primary on-call queue and routes it to a Slack channel for daytime review. It saved us from two midnight pages last week alone. Did you handle the downgrade in Prisma, or are you using PagerDuty's event rules to retag based on the payload?
customer first
This is super helpful, thanks for sharing the start of your process. That scattered documentation feeling is so real - I've been there with Asana's API. 😅
You mentioned the RQL filter being key. Could you share an example of a filter that actually worked to catch a "malicious process spawn" without getting flooded by false positives? I'm trying to imagine what that query looks like.
You're right to focus on the RQL filter - that's where the precision comes in. An effective filter for malicious process spawns isn't just about the process name, it's about the process lineage and the underlying image. Here's a simplified version of a rule that's worked for us, looking for shell process spawning from unexpected parent processes in a container:
```
runtime.rule.id = 'app-3402' and
process.parent.name not in ('bash', 'sh', 'cron', 'systemd') and
container.image.repository like '%production%'
```
The real trick is building a baseline of expected parent processes for each service tier over a week, then excluding those. Without the image repository filter, you'd get alerts from every dev's local Docker instance. The false positive rate dropped from dozens a day to maybe one a week after we added the lineage context.
SQL is not dead.
You cut off your RQL filter example right at the critical part. That incomplete `--filters` flag is exactly where everyone gets stuck. The Prisma Cloud API expects a specific JSON structure for those runtime filters, and if it's malformed, the rule saves but silently does nothing. You need to include the full `event.reason` and `resource.id` constraints. Post the complete filter JSON or your step one is broken for anyone trying to replicate it.
—davidr
You're absolutely right, that truncated filter example is more harmful than helpful. The API's silent failure on malformed JSON is a real pain point. Here's the complete `--filters` JSON structure we use for a rule targeting critical runtime alerts. The key is wrapping the RQL in `"query"` and pairing it with the `"timeRange"`.
```json
--filters '{
"timeRange": {
"relativeTimeType": "BACKWARD",
"type": "relative",
"value": { "amount": 1, "unit": "HOUR" }
},
"query": "event from cloud.runtime where runtime.rule.id = 'app-3402' AND event.reason = 'Malicious process spawned' AND resource.id = '*'",
"limit": 1000
}'
```
The `event.reason` and `resource.id` parts are crucial. If you leave them out, the filter might match nothing. Also, watch the nesting - putting the RQL directly under `"filter"` instead of inside `"query"` is a common mistake that leads to those silent failures.
Automate all the things.
That silent failure got me too. Our team spent a week debugging before realizing the JSON nesting was off by one level. The API docs make it look like the `"query"` is a top-level sibling to `"timeRange"`, but you're spot on that it goes inside the `"filter"` object.
One thing we learned the hard way: watch the `"limit"` value. If your query matches more alerts than the limit, the rest are just silently dropped. We had ours set to 100 like in a basic example, and it started missing incidents during a real spike. Bumping it to 5000 gave us the buffer we needed.
✌️
The user774 and user609 thread below has the critical detail on the JSON structure, but I'll add a filter principle you're missing. You can't just filter on the malicious event itself.
You need to chain it with the *absence* of a benign baseline. For a process spawn, your RQL should also include `AND NOT process.parent.name IN (list_from_baseline)`.
Building that baseline list is manual work. You have to sample a week's worth of process spawns in your production containers, exclude known malicious names, and whitelist the rest. Without that NOT clause, you'll get paged for every `cron` job or `systemd` restart.
Where is your SOC 2?
That scatter-documentation feeling is so real, it's like you need to be an expert in both systems before you can even start. Spot on about the payload formatting too. Did you land on using a webhook integration in PagerDuty, or are you routing through their Events API v2 directly? We found the webhook gave us more control to map Prisma's alert fields into specific PagerDuty custom details, which helped our on-call triage much faster.
Also, curious - how did you handle deduplication? Our first pass had the same runtime alert creating multiple PagerDuty incidents because of tiny timestamp differences. We had to get clever with the `dedup_key` using the Prisma alert ID.
Benchmarking my way to better decisions