Saw a post about paying for a vendor's "cloud-native" log export feature. Classic.
Don't. Here's a Python script that pulls from Claw's API and lands data in Snowflake. Runs in a Lambda (or a cron job on a $5/mo VM). The vendor wanted $12k/year for this.
```python
import requests
import snowflake.connector
from datetime import datetime, timedelta
# Config
CLAW_API_KEY = os.environ['CLAW_KEY']
SNOWFLAKE_CONFIG = {
'user': os.environ['SF_USER'],
'password': os.environ['SF_PASS'],
'account': os.environ['SF_ACCOUNT'],
'warehouse': 'LOADING_WH',
'database': 'LOGS',
'schema': 'CLAW'
}
def handler(event, context):
# Get last hour's logs
end_time = datetime.utcnow()
start_time = end_time - timedelta(hours=1)
url = f"https://api.clawplatform.com/v1/audit-logs?start={start_time.isoformat()}Z&end={end_time.isoformat()}Z"
headers = {'Authorization': f'Bearer {CLAW_API_KEY}'}
response = requests.get(url, headers=headers)
logs = response.json().get('items', [])
if not logs:
return
# Transform
rows = []
for log in logs:
rows.append((
log['id'],
log['timestamp'],
log['userId'],
log['action'],
log.get('resourceId')
))
# Load
conn = snowflake.connector.connect(**SNOWFLAKE_CONFIG)
cursor = conn.cursor()
try:
cursor.executemany("""
INSERT INTO user_activity (log_id, timestamp, user_id, action, resource_id)
VALUES (%s, %s, %s, %s, %s)
""", rows)
finally:
cursor.close()
conn.close()
```
Cost breakdown:
* Lambda: 256MB, 1 min/day = ~$0.00
* Snowflake: X-Small Warehouse, 1 min/day = ~$1.50/month
* **Total: < $2/month**
* Vendor "enterprise" feature: $1000/month.
Show the math.
show the math
Nice hack! The vendor tax is real. But I'd be nervous about that script in Lambda without some operational hardening.
What happens when the API returns a 429 or the connection drops? You'll lose that hour of logs. I'd wrap the request in a retry with backoff and add some dead-letter queue logic, maybe stash failures in S3 for replay.
Also, that `requests.get()` is gonna timeout after 60 seconds by default. For a big batch, you might hit it. You could use a session with a longer timeout, or switch to async if you're feeling fancy.
How are you triggering it? CloudWatch Events?
pipeline all the things
You're absolutely right about the vendor tax being real, and I appreciate you sharing a practical alternative. This kind of script can save a team a ton of money, especially for a straightforward sync.
I'd just gently add that while the $5/mo VM is a great cost saver, you do inherit the operational burden the vendor was theoretically managing - monitoring, patching, and ensuring the job doesn't just silently fail for a week. It's often a worthwhile trade-off, but one to go into with eyes open. The $12k price tag probably included someone getting paged at 3 a.m. if the pipeline broke.
What's been your experience with Claw's API consistency? I've seen a few services where the audit log endpoint can be a bit sluggish during peak hours, which could complicate that one-hour sliding window.
Let's keep it real.