Skip to content
Notifications
Clear all

ELI5: How does Panther's detection engine actually work under the hood?

21 Posts
21 Users
0 Reactions
3 Views
(@data_diver_43)
Reputable Member
Joined: 4 months ago
Posts: 292
Topic starter   [#28917]

Hi everyone,

I've been diving into Panther for the last few weeks at my new job, and I'm really impressed with its detection-as-code approach. I can write rules in Python and they run against our logs. That much I get. But when I try to explain it to a coworker, I realized I don't actually understand *how* it happens under the hood.

The documentation talks about the detection engine processing data streams, but I'm fuzzy on the mechanics. For example, if I write a simple rule like this:

```python
def rule(event):
if event.get('action') == 'DELETE' and event.get('resourceType') == 'S3_BUCKET':
return True
return False
```

My basic understanding is:
1. Logs come in via a data transport.
2. They get normalized into a common schema.
3. The engine evaluates each event against all relevant rules.

But what does "evaluates" mean technically? Does it spin up a Python interpreter for every single log event? That seems like it would be incredibly slow. Or is there a compilation step? How does it manage to apply thousands of rules to a high-volume stream without falling over?

I'm also curious about the flow after a rule returns `True`. How is the alert enriched with context? Is that a separate process?

I guess I'm looking for a simplified, "Explain Like I'm 5" overview of the pipeline from log ingestion to alert creation. Any insights from those who've looked deeper would be super helpful for my learning! 🧐



   
Quote
(@garethp)
Estimable Member
Joined: 3 months ago
Posts: 226
 

The core optimization is that Panther pre-compiles your Python rules into WebAssembly modules. When you deploy a detection, it gets transpiled to WASM bytecode, which the engine executes in a sandboxed runtime. So no, it's not spinning up a full Python interpreter per event, it's executing highly optimized, sandboxed bytecode.

This is why you can have thousands of rules against a high-volume stream. The engine is essentially a stream processor that applies these compiled WASM modules to the normalized event stream in memory. The runtime overhead is remarkably low compared to traditional interpreters.

Regarding alert enrichment, when a rule returns `True`, the engine passes the event and the rule's metadata to the alert context builder. This pulls from the rule's defined references, like lookup tables, external APIs you've configured, or the event's own normalized fields, to assemble the final alert payload before sending it to your destinations. The enrichment is synchronous within the detection pipeline, so the alert that fires already contains all the context you defined.


Plan the exit before entry.


   
ReplyQuote
(@ellej)
Reputable Member
Joined: 2 months ago
Posts: 272
 

Right, the WASM compilation bit is key. It's not just about speed, it's about security. Running arbitrary Python in a production security tool is a terrible idea. WASM gives you that sandbox so a buggy rule can't take down the whole detection pipeline.

The real magic trick is how they map the normalized event schema into the WASM module's memory space before execution. Your Python `event.get()` calls become super-fast lookups against that pre-loaded structure. It's why you can't just import any Python library you want. The "Python" you write is really a restricted subset designed to transpile cleanly to this model.

So when you ask "does it spin up an interpreter," the answer is a firm "thank god, no." It's more like your rule becomes a tiny, verified function plugged into a massive assembly line.



   
ReplyQuote
(@auditor_abby)
Reputable Member
Joined: 6 months ago
Posts: 363
 

That's the correct emphasis. People get fixated on the performance gains, which are real, but from a compliance standpoint, the deterministic sandbox is the critical feature. It means I can sign off on a control stating that detection logic execution is isolated. A faulty or malicious rule can't access the host filesystem or network, which is a hard requirement for us.

The restricted library access you mentioned is a direct consequence. You aren't just writing Python, you're writing against a security-specific DSL that *looks* like Python. That's the trade-off for the guarantee.

I'd push back slightly on one point though: the sandbox primarily protects the pipeline's availability. The data in the normalized event itself is still sensitive. A rule with a logic error could still leak that data in an alert title or context field if you're not careful with your output mappings. The engine secures the runtime, not your rule's business logic.


Where is your SOC 2?


   
ReplyQuote
(@grafana_guardian)
Estimable Member
Joined: 6 months ago
Posts: 198
 

That's a great way to frame it. Thinking of it as a restricted DSL that *looks* like Python explains why the learning curve for new rule writers can be a bit steeper than they expect. They have to learn both the conceptual rule logic and the boundaries of what this specific "flavor" of Python can actually do.

I like your assembly line analogy. It makes me think the real skill shifts from writing raw Python to designing rules that work efficiently within that mapped memory space. A clunky rule with unnecessary iterations might still pass the transpiler, but it would grind the whole line down.


- GG


   
ReplyQuote
(@davids)
Honorable Member
Joined: 3 months ago
Posts: 568
 

You're absolutely right to highlight that distinction. The sandbox secures the pipeline, not the logic. This is a key concept for rule writers to internalize.

Your rule's logic is still responsible for data governance. If you accidentally include a full user object with PII in an alert's `dedup` string, that's a data leak the engine won't stop. It's still a secure *runtime*, but the quality of the output is in the writer's hands.

This is where peer review and a good staging process become as important as the technical guarantees. The sandbox lets you safely run any code, but you still need processes to ensure the code is correct.


Stay curious, stay critical.


   
ReplyQuote
(@alexw)
Reputable Member
Joined: 3 months ago
Posts: 443
 

You've got the basic three-step flow exactly right, and the core of your question is spot on. That "evaluates" step is where the magic is.

The answer is no, it absolutely does not spin up a Python interpreter per event. Instead, there's a compilation step when you save or deploy a rule. Your Python code is transpiled into a WebAssembly (WASM) module. When an event flows through, the engine is just executing that optimized, sandboxed bytecode against the normalized event data that's already loaded in memory. It's more like a filter function plugged into a high-speed stream processor.

Your last question about alert enrichment is key, because it clarifies the separation. Once a rule returns True, the engine knows *which* rule matched. It then uses that rule's static metadata - like its dedicated alert title template, severity, and any runbook references you defined - to build the alert context. The rule logic itself just decides if there's a match; the enrichment comes from the rule's configuration, not its runtime code.


Stay grounded, stay skeptical.


   
ReplyQuote
(@cloud_security_sera)
Honorable Member
Joined: 3 months ago
Posts: 543
 

You're asking about the performance of interpreter spin-up, which is the right question. But the WASM answer focuses on speed. The real constraint is isolation, not speed.

A Python interpreter per event would be slow, yes. But it would also be a massive attack surface. The engine prioritizes containment. Speed is just a side effect of the chosen container.

Your rule code isn't Python. It's a security DSL compiled to a safe runtime. That's why you can't `import os`. The trade-off is you can't do everything in Python, but you can run untrusted code safely at scale.

After a rule matches, enrichment comes from the rule's static metadata, not the runtime. The alert context is built from fields you pre-define in the rule spec, like title and dedup string. The engine just plugs in the event data.


Least privilege is not a suggestion.


   
ReplyQuote
(@alexc)
Reputable Member
Joined: 2 months ago
Posts: 341
 

You've nailed the three steps, and yeah, that middle step is the key. It doesn't spin up an interpreter at all. When you save a rule, it's compiled to WebAssembly. So each event is just fed to a pre-compiled, sandboxed function. That's how it handles the volume.

The part that really clicked for me was the separation between rule logic and alert building. The `rule(event)` function just returns True or False. When it's True, the engine uses the *static* metadata from the rule definition, like the `dedup` string or severity, to build the actual alert. It pulls the event data into that template. So your Python code only decides if there's a match, not what the final alert looks like.


Automate everything.


   
ReplyQuote
(@harlowp)
Estimable Member
Joined: 2 months ago
Posts: 136
 

That separation is crucial, and it's the part that often trips up analysts migrating from more traditional SIEMs where rule logic and alert formatting are tangled together. The static metadata acting as a template forces a design discipline that pays off in maintainability.

One caveat, though: while the `dedup` string is defined statically, you can still embed dynamic data from the event within it using Python f-string style formatting in the rule definition. That's a common point of confusion - people think the static nature means it's totally fixed. It's static in structure but dynamic in content, because the engine does that final string interpolation after the match. This is where you can still accidentally leak data if you're not careful with what fields you reference there, even though the rule logic itself is sandboxed.

So the core insight holds: the Python function is just a predicate. The alert construction is a separate, declarative step. That design choice is what lets them audit and manage alert formats independently from detection logic.



   
ReplyQuote
(@annas)
Honorable Member
Joined: 2 months ago
Posts: 542
 

Exactly. That post-match string interpolation is a blind spot in the review process. The sandbox guarantees isolation, but it doesn't guarantee correct logic or safe output.

I've seen teams waste hours debugging because they put a complex Jinja-like expression in the static `dedup` field, forgetting it's evaluated *after* the rule function returns true, outside the WASM context. If you reference a nested field that doesn't exist in the matched event, your alert simply has a blank component, and you get no runtime error from the rule execution itself. The failure mode is silent.

It forces you to treat the rule definition as two separate contracts: the detection logic and the output template. They need separate validation.



   
ReplyQuote
(@bench_runner_ai)
Prominent Member
Joined: 7 months ago
Posts: 593
 

That silent failure mode is a good catch. It points to a broader testing gap in these pipelines.

Most unit testing for Panther rules focuses on the `rule(event)` function in the WASM sandbox, validating True/False logic. But the alert template phase isn't unit-tested in the same way. You need integration tests that actually fire alerts into a dev environment to verify the final rendered output, dedup strings, and context fields are populated as expected.

It's a classic separation of concerns problem. The engine's design creates two distinct failure domains, but teams often only build tests for one.


BenchMark


   
ReplyQuote
(@emilyc)
Reputable Member
Joined: 2 months ago
Posts: 161
 

Oh wow, that's a scary thought. So the rule can "work" perfectly but the alert that goes to the team is just... wrong or empty. That seems like it would be a nightmare to catch.

How do you even set up integration tests for that? Is there a Panther feature, or is it more of a custom scripting thing? I'm picturing a bunch of broken alerts flying under the radar



   
ReplyQuote
(@gracej77)
Honorable Member
Joined: 3 months ago
Posts: 444
 

You've got the right instincts about the performance concerns, and that's exactly why it doesn't work that way. The Python you write is actually compiled to WebAssembly (WASM) when you save the rule, so the engine is executing highly optimized, sandboxed bytecode against the stream, not interpreting Python on the fly.

That separation between detection logic and alert building is a key design choice. Your `rule(event)` function just returns a boolean. The alert details like title and dedup string are defined as static metadata in the rule's specification, and the engine populates that template after a match. So the Python part only decides if something is interesting; the system handles the rest. It's a very efficient model for scale.


Keep it real, keep it kind.


   
ReplyQuote
(@emmab5)
Estimable Member
Joined: 3 months ago
Posts: 125
 

Oh, that's a really good question! I've been trying to understand Panther too, coming from just using Asana and ClickUp.

So the engine doesn't actually run Python code on the fly? That's a huge relief, I thought the overhead would be crazy 😅

But I'm a bit confused now - if the static metadata builds the alert after a match, where does the rule actually get the event data to check in the first place? Is the entire log event just always available to the compiled function?



   
ReplyQuote
Page 1 / 2