I've been using Lacework for compliance monitoring across a few AWS accounts, and while the dashboard is great for a high-level view, I needed a way to track specific control failures over time. I wanted to see which issues were new, which ones we fixed, and if any regressed month-to-month.
The API is pretty comprehensive, so I wrote a Python script that fetches the last two compliance reports and diffs them. It outputs a simple markdown table. This is super useful for our internal review meetings and for proving progress to auditors.
Here's the core part of the script. You'll need to set up your Lacework API credentials.
```python
import pandas as pd
from datetime import datetime, timedelta
import laceworksdk
# Initialize client
lw_client = laceworksdk.LaceworkClient()
def get_compliance_report(days_back):
end_time = datetime.now() - timedelta(days=days_back)
start_time = end_time - timedelta(days=30)
response = lw_client.compliance.get(
"AWS_CIS_S3",
start_time=start_time,
end_time=end_time
)
# Parse response into a list of violations
violations = []
for recommendation in response['data']:
if recommendation['status'] == 'FAILED':
violations.append({
'account': recommendation['account_alias'],
'violation_id': recommendation['violation_id'],
'violation_desc': recommendation['violation_desc'][:100] # truncate
})
return pd.DataFrame(violations)
# Get last month's and current month's reports
df_previous = get_compliance_report(30)
df_current = get_compliance_report(0)
# Find new failures and resolved ones
new_failures = df_current[~df_current['violation_id'].isin(df_previous['violation_id'])]
resolved = df_previous[~df_previous['violation_id'].isin(df_current['violation_id'])]
# Print a summary table
print("| Status | Count | Example Violation ID |")
print("|---|---|---|")
print(f"| New Failures | {len(new_failures)} | {new_failures.iloc[0]['violation_id'] if len(new_failures) > 0 else 'N/A'} |")
print(f"| Resolved | {len(resolved)} | {resolved.iloc[0]['violation_id'] if len(resolved) > 0 else 'N/A'} |")
```
The script helps us move faster. Instead of manually comparing PDF exports, we can now:
* Automate the diff as part of our monthly security review
* Track trends for specific accounts or services
* Feed the data into our internal dashboards
One gotcha: the Lacework API date ranges are based on the assessment time, not the violation occurrence. Make sure your `days_back` parameter aligns with when your reports actually run.
Has anyone else built similar tooling around Lacework data? I'm curious if you're pulling compliance data into a warehouse or using their alerts for something like automated Jira ticket creation.
terraform and chill
This is such a great approach for tracking compliance drift! I used a similar diff script during a major Salesforce migration to track custom object adoption over sprints. One thing I'd add - you might want to build in a small "grace period" overlap for your date ranges. APIs can sometimes have timestamp quirks, and you don't want to miss a violation that was remediated right at the report boundary. Maybe pull 35 days back for the older report, but still compare it as a 30-day window? Just a thought from painful experience!
Also, for your review meetings, do you snapshot the diff output somewhere? I started saving mine to a simple database table with a run timestamp, which let me build a trend line of "new issues per month." That metric became gold for our change management folks.
migrate with care
You're building a dashboard to track a dashboard's data. Why?
That's a sign your main tool isn't giving you what you need.
Every script like this becomes a liability:
* API breaks, script breaks.
* Someone leaves, nobody knows the script.
* You're now responsible for its output and security.
Push back on Lacework. Tell them you need this as a core feature. If they can't provide a simple compliance trend report, you're paying them for half a product.
Until then, at least store the raw JSON responses with timestamps. Your pandas logic will change; the raw data won't.
Simplicity is the ultimate sophistication
You're absolutely right about the liability, and I've lived through that pain. An API change can turn a critical reporting script into a weekend firefight.
But here's my counter, from someone who's done this across three different SaaS platforms: even if Lacework adds a trend report tomorrow, it likely won't map exactly to our internal review format or tie into our change tickets. There's always a gap between the vendor's dashboard and the specific metrics your team actually uses to make decisions.
The key, which you nailed, is treating the raw API data as the only artifact you keep. The script that transforms it is disposable. I version-control mine alongside a simple schema file, and the CI job that runs it archives the raw JSON with a timestamp. If the script breaks, you just lose one month's diff, not the underlying history.
Still, you've got me thinking. Maybe the real solution is a lightweight, vendor-neutral orchestration layer that just fetches and stores the data. Then you can plug in different transformation scripts as needed without ever touching the raw archive.
Always have a rollback plan.
Nice approach with the pandas diff. I've done something similar for PCI reports, but found the raw API responses can be huge. Pulling the last two months for multiple accounts started hitting timeouts.
I switched to using their async endpoints and storing the raw JSON in S3 first, then running the diff as a separate step. Takes a bit more setup, but it's more reliable for a larger dataset.
Have you run into any performance issues when the violation list gets long?
Latency is the enemy, but consistency is the goal.