Deployed Panther for 50 AWS accounts - unexpected issues

After evaluating several SIEM options for a multi-account AWS environment, my team decided to proceed with Panther. The promise of serverless, scalable log analysis aligned well with our need for a centralized security data lake across 50 member accounts. The deployment via Terraform was largely successful for the core infrastructure.

However, we encountered several friction points at scale that weren't apparent in the single-account POC.

**1. S3 Bucket Proliferation & Naming Collisions**
Each Panther module (e.g., `panther-log-analysis`, `panther-cloud-security`) creates its own S3 bucket in the master account. With our multi-account setup, the automated creation of cross-account IAM roles and KMS key policies led to AWS Service Catalog provisioning errors due to S3 bucket name uniqueness constraints. We had to implement a custom prefixing strategy, which required modifying the Terraform modules.

**2. Real-Time Updates via SQS Queues**
The documented method for real-time updates to detection policies uses SQS. At our scale, with hundreds of rules and a high volume of log traffic, we observed significant latency (5-7 minutes) between a policy change and its application. The SQS queue processing couldn't keep pace, causing a delay in critical security updates. We are now investigating a hybrid approach using S3 notifications directly to Lambda.

**3. Cost Attribution Complexity**
While Panther itself is serverless, the AWS cost footprint is distributed across:
- CloudWatch Logs subscriptions (ingestion)
- S3 storage & PUT requests (data lake)
- Athena/Glue queries (scheduled searches & data exploration)
- Lambda invocations (rule execution)

Attributing these costs back to individual source accounts or business units is non-trivial. We've had to build additional tagging automation and CUR reports to provide visibility.

Here is a snippet of the Terraform override we used for the S3 bucket naming issue:

```hcl
module "panther_log_analysis" {
source = "panther-community/panther-aws-bootstrap/aws"
version = "~> 1.0"

# ... other config
s3_bucket_name_override = "ourcompany-panther-log-${var.region}-${random_id.suffix.hex}"
}
```

Overall, Panther is processing several TB of logs daily effectively, and the Python-based rule writing is excellent. The issues were primarily around the operational overhead of a large, multi-account deployment rather than the core analytics engine. I'm interested to hear if others have hit similar scaling limits and how you tuned the data pipeline.