Skip to content
Notifications
Clear all

Our team built a custom Slack bot to summarize daily Lacework alerts. Sharing the script.

6 Posts
6 Users
0 Reactions
80 Views
(@cloud_cost_optimizer)
Honorable Member
Joined: 8 months ago
Posts: 473
Topic starter   [#10154]

Our team recently completed a quarterly cloud cost and security audit, which included a deep analysis of our Lacework spend relative to alert volume and operational overhead. A key finding was that while Lacework provides comprehensive alerting, the signal-to-noise ratio for our Kubernetes and AWS environments led to alert fatigue, causing critical issues to be missed in a high-volume Slack channel.

To address this, we engineered a cost-effective internal solution: a Slack bot that consumes, aggregates, and summarizes Lacework alerts on a daily cadence. This has reduced our daily alert volume in the primary channel by approximately 70%, allowing the SOC to focus on the summarized report and high-severity items. Below is the architectural breakdown and the core processing script.

**Architecture Overview:**
* **Event Source:** Lacework is configured to send all alerts to an AWS SQS queue via its native AWS integration.
* **Processing Engine:** A Python-based AWS Lambda function, triggered hourly, fetches and batches messages from SQS.
* **Data Aggregation:** The function parses the Lacework alert JSON, categorizes alerts by severity, resource type, and event subtype.
* **Storage:** Summary counts and unique alert IDs are stored temporarily in Amazon DynamoDB (with a TTL of 36 hours) to enable daily roll-ups.
* **Reporting:** A second scheduled Lambda function runs daily at 08:00 UTC. It queries DynamoDB for the prior 24-hour period, generates the summary, and posts to Slack via a webhook.

**Core Aggregation Lambda Logic (Python Excerpt):**
The following snippet shows the key aggregation logic within the hourly processor.

```python
import json
import boto3
from collections import defaultdict
from datetime import datetime, timedelta

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('LaceworkAlertAggregates')

def aggregate_alert(lacework_alert):
"""Extracts and structures key fields for aggregation."""
alert_id = lacework_alert.get('ID')
severity = lacework_alert.get('SEVERITY', 'Unknown')
event_subtype = lacework_alert.get('EVENT_SUBTYPE', 'Unknown')
resource_type = lacework_alert.get('RESOURCE_TYPE', 'Unknown')

# Create a composite key for grouping
aggregate_key = f"{severity}#{resource_type}#{event_subtype}"
return aggregate_key, alert_id

def lambda_handler(event, context):
aggregated_data = defaultdict(lambda: {'count': 0, 'alert_ids': set()})

for record in event['Records']:
body = json.loads(record['body'])
for alert in body.get('DATA', []):
agg_key, alert_id = aggregate_alert(alert)
aggregated_data[agg_key]['count'] += 1
aggregated_data[agg_key]['alert_ids'].add(alert_id)

# Write aggregates to DynamoDB
for agg_key, data in aggregated_data.items():
table.put_item(
Item={
'AggregateKey': agg_key,
'DateHour': datetime.utcnow().strftime('%Y-%m-%d-%H'),
'AlertCount': data['count'],
'AlertIds': list(data['alert_ids'])[:10], # Store sample IDs
'TTL': int((datetime.utcnow() + timedelta(hours=36)).timestamp())
}
)
```

**Cost & Operational Impact:**
* **Development Cost:** < 2 engineer-weeks.
* **Runtime Cost:** ~$3.10/month (Lambda: 2M invocations @ $0.20, DynamoDB: ~$2.90).
* **Outcome:** The daily summary post provides a clear trendline of alert categories, enabling us to tune Lacework policies proactively. We've subsequently identified and eliminated several persistent low-severity rules, which is projected to reduce our Lacework data ingestion costs by an estimated 15% next quarter.

This approach exemplifies a FinOps principle: optimizing not just the direct cloud spend, but the operational cost associated with managing cloud resources. The script is now a staple in our internal toolkit for managing security observability.

-cc


every dollar counts


   
Quote
(@bench_beast)
Noble Member
Joined: 5 months ago
Posts: 723
 

Can you share the alert deduplication logic? The raw volume isn't the only problem, duplicate alerts for the same underlying event are a major drain. Our own script had to implement a time-windowed hash based on resource ID and alert title to cut another 30% of noise.

Also, what's the average latency from Lacework event to Slack summary? We saw a 2-3 hour lag with SQS/Lambda batching, which was acceptable for daily digests but not for criticals.


Benchmarks don't lie.


   
ReplyQuote
(@grafana_guy_night)
Honorable Member
Joined: 8 months ago
Posts: 427
 

That 70% reduction is huge! Congrats on the build. I've been trying to cut down Prometheus alert noise, so seeing this approach is really helpful.

Could you share the categorization logic? Like, do you group by something like `account:resource_type:alert_type` or is it more detailed? Trying to figure out how you got it so clean.

Also, what do you do with the critical alerts? Do they still go straight to the main channel while the bot handles the summary?



   
ReplyQuote
(@charlieg)
Honorable Member
Joined: 4 months ago
Posts: 503
 

That 70% reduction sounds impressive until you ask how they're measuring it. Is that just raw alert counts, or does it factor in that you might be hiding important context in the summary? Aggregation can smooth over spikes that actually matter.

For categorization, "account:resource_type:alert_type" is a decent start, but it's the exceptions that kill you. What about the same alert type across three microservices in one pod? Does that get rolled up into one line item and lose all granularity? I've seen teams miss a brewing incident because the "clean" summary buried the affected resource count.

And on critical alerts, if they're still firing directly to the main channel, you haven't solved alert fatigue. You've just moved the medium-priority noise. The real test is whether the bot's summary is the *only* thing hitting that channel, criticals included, with a sane triage workflow attached. Otherwise you've just built a fancy filter for stuff you're already ignoring.


cg


   
ReplyQuote
(@docker_diver)
Honorable Member
Joined: 5 months ago
Posts: 496
 

Cool approach with SQS and Lambda. But I'm still wrapping my head around parsing the Lacework JSON. Could you share a snippet of how you map their alert fields to your categories? I tried something similar and got stuck on all the nested event details.


Containers are magic, but I want to know how the magic works.


   
ReplyQuote
(@katherineh)
Eminent Member
Joined: 4 months ago
Posts: 30
 

The nested JSON is the real challenge. Lacework's schema varies by alert source, so you need conditional logic. For our mapping, we treat the `eventDetails` object as the primary source, extracting a normalized key from a combination of `eventId`, `resourceType`, and a cleaned `alertTitle`.

Here's a simplified version of our categorization function. It handles the common case of AWS Config alerts and the more complex nested structures for Kubernetes events.

```python
def categorize_alert(alert_json):
# Start with the alert type as a fallback
category = alert_json.get('alertType', 'UNKNOWN')

# Check for AWS Config pattern
if 'eventDetails' in alert_json and 'resourceType' in alert_json['eventDetails']:
res_type = alert_json['eventDetails']['resourceType']
# Example: Map 'AWS::EC2::SecurityGroup' to 'EC2 Security'
if res_type.startswith('AWS::'):
category = res_type.split('::')[1] + ' Events'

# Special handling for K8s, where details are nested deeper
elif alert_json.get('alertSource') == 'KubernetesAuditLog':
# The entity map often contains the pod/service info
entities = alert_json.get('entityMap', {})
if 'pod_name' in entities:
category = 'K8s Pod: ' + entities.get('namespace', 'default')
return category
```

The critical step is logging a sample of unmapped alerts to iteratively build out your logic. We started with about 20 categories and expanded to over 50 as we ingested real data.


—KH


   
ReplyQuote