Hi everyone! I'm new to the GRC side of things and ran into a problem with audit log exports. 😅
Our security team needs 90 days of logs for a SOC 2 review, but the vendor API only lets me pull 30 days at a time. I tried a script to loop through months, but I'm stuck on handling pagination and the date ranges correctly. Is this a common limitation? How do you all handle it?
Here's my basic attempt so far (Python):
```python
import requests
from datetime import datetime, timedelta
# This only gets the last 30 days
response = requests.get('https://api.example.com/v1/audit_logs',
params={'since_days': 30},
headers={'Authorization': 'Bearer TOKEN'})
logs = response.json()
```
Any advice on a better approach or if I should push back on this limit? Thanks in advance for your help!
You're asking if it's common? Absolutely. Vendors love these artificial limits because they push you toward their premium "enterprise" tiers or expensive log forwarding services. Check if they even store 90 days by default or if that's another upsell.
That script approach is on the right track, but you're going to hate dealing with their pagination tokens and rate limits. And watch out for date overlaps when you loop.
Pushing back is your best move. Ask their support for the official data retention policy in writing, then ask why their API doesn't align with it. For SOC 2, you'll need to document this as a limitation in your controls anyway.
cg
Totally feel your pain, it's such a frustratingly common wall to hit! Your script is the right starting point, but you'll need to swap `since_days` for actual start/end date parameters, like `start_time` and `end_time` in ISO format. Then loop in 30-day chunks.
Watch out for the pagination - you'll likely need to check for a `next_cursor` or `page_token` in the response and keep fetching until it's empty. And yeah, overlaps can mess up your totals, so make your chunks like "day 0 to 29, day 30 to 59."
It's a hassle, but doable for a one-time pull. For ongoing needs, I'd push back hard while you build the script. Good luck