While Lacework provides a comprehensive view of cloud activity, its data ingestion model can lead to significant and often surprising monthly costs, particularly for organizations with high-volume AWS environments. A substantial portion of this ingested data originates from AWS CloudTrail management events, which are notoriously verbose and contain a high frequency of low-signal, routine operations. Proactively filtering these events before they reach Lacework's pipeline is one of the most effective cost-control levers available.
The core strategy involves deploying an intermediary filtering mechanism, such as an AWS Lambda function, within the CloudTrail-to-Lacework data flow. This function inspects each event and applies a deterministic allowlist or blocklist based on event characteristics, forwarding only high-value events to the Lacework resource group or S3 bucket configured for ingestion. The key is to identify events that are redundant for security posture or threat detection.
A critical first step is profiling your current CloudTrail volume to establish a baseline. You can use Athena or a simple script against your S3 logs to categorize events. You will typically find a small number of event names constitute the majority of the log volume. Common candidates for filtering include:
* **`Describe` and `List` API calls:** Events like `DescribeInstances`, `ListBuckets`, and `ListFunctions` are executed constantly by consoles, scripts, and monitoring tools. They reflect read-only information gathering and rarely indicate a compromise on their own.
* **Console sign-in events:** While the first sign-in of a session is critical, subsequent `ConsoleLogin` events generated by session renewal can be filtered after the first instance per session.
* **Expected, automated noise:** Events from trusted infrastructure-as-code tools (e.g., specific `CloudFormation` events) or trusted service-linked roles performing routine health checks.
Below is a conceptual Python snippet for a Lambda filter focusing on blocklisting high-volume, low-value event names. This would be deployed to process events from the CloudTrail S3 bucket or directly from a CloudTrail log group subscription.
```python
import json
import gzip
import base64
# Define a set of event names to block. This list must be curated from your baseline.
BLOCKLISTED_EVENTS = {
'DescribeInstances',
'ListBuckets',
'GetObject',
'ListFunctions',
'ConsoleLogin', # Apply careful session logic for this one
'HeadBucket'
}
def lambda_handler(event, context):
output = []
for record in event['records']:
payload = base64.b64decode(record['data'])
# Decompress if from S3 or Kinesis Firehose
decompressed_payload = gzip.decompress(payload).decode('utf-8')
cloudtrail_data = json.loads(decompressed_payload)
filtered_records = []
for ct_event in cloudtrail_data.get('Records', []):
event_name = ct_event.get('eventName')
# Core filtering logic: only forward if NOT in blocklist
if event_name not in BLOCKLISTED_EVENTS:
filtered_records.append(ct_event)
# If any records remain after filtering, prepare them for forwarding
if filtered_records:
new_payload = {'Records': filtered_records}
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': base64.b64encode(json.dumps(new_payload).encode('utf-8'))
}
output.append(output_record)
else:
# Drop the entire original record if all events were filtered
output.append({
'recordId': record['recordId'],
'result': 'Dropped',
'data': base64.b64encode(b'[]')
})
return {'records': output}
```
Implementing this requires a rigorous validation period. You must forward the filtered event stream to a test Lacework resource group and compare alerts and the Polygraph data model against your production unfiltered feed for at least two weeks. The goal is to confirm no degradation in detection of suspicious `Create`, `Delete`, `Modify`, `Put`, or `Terminate` events. The cost savings, however, are immediate and can often reduce CloudTrail-related ingestion volume by 40-60%, directly impacting the bottom line without materially affecting security coverage.
Totally agree on profiling first. I've seen teams skip that and filter too aggressively, then miss weird IAM failures that were actually important.
Instead of a Lambda, you could also run a Fluent Bit daemonset in EKS if you're already shipping logs through the cluster. You can tag and drop events right at the source with filters, keeps the architecture a bit simpler. Here's a basic filter I've used for CloudTrail:
```yaml
[FILTER]
Name grep
Match cloudtrail.*
Regex eventName ^(Describe|List|Get).*
Exclude On
```
That'll strip out all the read-only calls. You can pipe the remaining stream directly to Lacework's endpoint. Cuts like 60% of volume in most accounts I've worked on.
Have you played with OpenTelemetry collectors for this? Might be overkill but the transformation capabilities are pretty solid.
YAML is not a programming language, but I treat it like one.
> strip out all the read-only calls
That's a solid starting filter, but wouldn't you still want to keep some failed read calls? A DescribeInstances that's denied could be a recon attempt, but dropping all the successes cuts so much noise. Have you found a reliable way to keep the failures from those APIs while still filtering the successes?
Profiling the baseline volume is indeed the only sane way to start. Skipping that step turns cost optimization into a security gamble. When using Athena for that initial analysis, I always extend the query to include the `errorCode` field and count `ReadOnly` operations separately from `Write` operations. This gives you a clear matrix: high-volume/low-risk (successful reads), low-volume/high-signal (failures of any kind), and the critical write events.
One caveat with the Lambda intermediary approach is the need for rigorous error handling and an audit trail for the filter itself. If the function fails or has a logic error, you're silently dropping all data. You must implement dead-letter queues and log the count of filtered events per invocation, ideally with a sample, to confirm your logic is working as intended. This adds complexity but is non-negotiable for a production control.
Trust but verify