TIL: You can use Panther's Python SDK to backtest detection rules.

While reviewing Panther's documentation for an upcoming SOC 2 compliance push, I made a discovery that fundamentally shifts how I think about developing and hardening detection-as-code. Like many, I had relegated Panther's Python SDK (`panther_sdk`) to the role of a local testing utility—a handy way to run a rule against a few static logs. However, its true power lies in systematic backtesting, a practice I previously associated only with expensive, dedicated SIEMs or custom data pipelines.

The `panther_sdk` provides a `backtest` command that, when pointed at a detection and a historical data source, executes the rule logic against every record. This isn't just a simple pass/fail; it generates a detailed report. The key is that it operates on your actual data warehouse (Snowflake, BigQuery, etc.), not a synthetic subset. This allows you to measure a rule's historical efficacy and, more importantly, its signal-to-noise ratio *before* promotion to production.

Consider this workflow for a new detection rule designed to spot anomalous S3 bucket policy changes:

```python
# anomaly_s3_bucket_policy.py
from panther_base import lookup_aws_account_name
from panther_oss_helpers import deep_get

def rule(event):
# Detection logic
return (
event.get("eventName") == "PutBucketPolicy"
and event.get("userIdentity", {}).get("invokedBy") is None
and deep_get(event, "userIdentity", "sessionContext", "sessionIssuer", "userName")
not in ALLOWED_ROLES
)

def title(event):
return f"Suspicious S3 Bucket Policy Change in {lookup_aws_account_name(event.get('recipientAccountId'))}"
```

To backtest this against the last 90 days of CloudTrail data, you'd execute:
```bash
panther-sdk backtest
--path ./detections/anomaly_s3_bucket_policy.py
--data-source AwsCloudTrail
--start 2024-01-01
--end 2024-03-31
--output ./backtest_results/
```

The generated report provides critical, quantitative insights:
* **Total Matches:** How many times would this rule have fired? A number in the thousands suggests an overly broad rule.
* **Sample Alert Events:** You can inspect a subset of the matches to verify logic correctness.
* **Run Statistics:** Confirms the scan covered the intended time range and data volume.

This process exposes several classes of defects early:
* **False Positive Tuning:** If the rule fires on expected, automated infrastructure changes (e.g., from your Terraform pipeline), you must refine the exclusion logic (`ALLOWED_ROLES`).
* **Logic Gaps:** You might discover that certain risky scenarios (e.g., policy changes from a compromised console user) are missed because `invokedBy` is populated differently.
* **Performance Implications:** A rule that performs complex lookups on a high-volume log source might be inefficient; you can optimize before it impacts your scheduled queries.

Adopting this as a pre-commit gate in our detection CI/CD pipeline has been transformative. We no longer promote rules based solely on theoretical threat models. Instead, we require a backtest report demonstrating a manageable alert volume and clear examples of true positive detection over a significant historical period. This moves detection engineering from an art to more of a data-driven science, and it leverages the existing power of your data lake without additional infrastructure. The SDK handles the query generation and execution against your warehouse, making what would be a significant engineering project in other platforms a standard operational procedure.

--from the trenches