Alright, let's see if the collective wisdom here can help me stop drowning in alerts. We've been running Panther for about six months, primarily for log analysis on our AWS estate. The detection engine itself is fine, conceptually. The problem is that it's become the boy who cried wolf, specifically around our CI/CD pipelines.
Every time a Jenkins job fails, a CodeBuild run times out, or a deployment rolls back, Panther lights up like a Christmas tree. We've got rules flagging things like `ERROR` logs in CloudWatch from our build containers, IAM assumed role sessions from the CI system that get deemed "anomalous," and even our own deployment tooling triggering "Suspicious CloudTrail Events." I'm spending more time triaging and closing false positives than actually reviewing legitimate security events.
I've tried the obvious adjustments: refining the rule thresholds, adding more specific filters to exclude our CI service accounts and known IP ranges. But the nature of CI/CD is chaotic—temporary resources, dynamic error states, retries. It feels like I'm playing whack-a-mole.
Here's a simplified version of a rule that's been particularly noisy, catching any `AssumedRole` from our CI user but also, inexplicably, from our own break-glass admin role when engineers use it. The deduplication logic isn't helping as much as I'd hoped.
```python
def rule(event):
# Intention: Alert on any AssumedRole outside of our normal service accounts
if event.get('eventName') != 'AssumeRole':
return False
principal_arn = event.get('userIdentity', {}).get('arn', '')
target_role = event.get('requestParameters', {}).get('roleArn', '')
# CI service account is allowed
if 'ci-deployer-role' in principal_arn:
return False
# This is where it falls apart - our admin role is named 'admin-access'
# but CI sometimes assumes roles with 'target' in the name during deployments
# This is flagging both, incorrectly.
if 'admin-access' in target_role or 'target' in target_role:
# Should be fine, but we want to alert
return True
return False
def title(event):
return f"Suspicious AssumeRole by [{event.get('userIdentity',{}).get('arn')}]"
```
My current theory is that we need a more fundamental shift in approach. Perhaps:
* Creating a dedicated "CI/CD Noise" log source with its own, heavily filtered, rule set?
* Leveraging Panther's alert context or severity to dynamically downgrade these, but that feels like sweeping the problem under the rug.
* Writing rules that are more stateful, acknowledging that a failed build often leaves a trail of related errors that should be grouped into a single incident.
What are the rest of you doing? Is there a pattern for building a "forgiving" rule set for known-noisy environments like CI/CD, without completely blinding yourself to a real attack that might leverage those same pathways? I'm particularly interested in how you're structuring rule logic to account for the temporal and linked nature of pipeline events.
-- old salt